×

zbMATH — the first resource for mathematics

A game-based abstraction-refinement framework for Markov decision processes. (English) Zbl 1233.90276
Summary: In the field of model checking, abstraction refinement has proved to be an extremely successful methodology for combating the state-space explosion problem. However, little practical progress has been made in the setting of probabilistic verification. In this paper we present a novel abstraction-refinement framework for Markov decision processes (MDPs), which are widely used for modelling and verifying systems that exhibit both probabilistic and nondeterministic behaviour. Our framework comprises an abstraction approach based on stochastic two-player games, two refinement methods and an efficient algorithm for an abstraction-refinement loop. The key idea behind the abstraction approach is to maintain a separation between nondeterminism present in the original MDP and nondeterminism introduced during the abstraction process, each type being represented by a different player in the game. Crucially, this allows lower and upper bounds to be computed for the values of reachability properties of the MDP. These give a quantitative measure of the quality of the abstraction and form the basis of the corresponding refinement methods. We describe a prototype implementation of our framework and present experimental results demonstrating automatic generation of compact, yet precise, abstractions for a large selection of real-world case studies.

MSC:
90C40 Markov and semi-Markov decision processes
91A15 Stochastic games, stochastic differential games
90C15 Stochastic programming
Software:
CEGAR; PASS; PRISM
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Baier C, Kwiatkowska M (1998) Model checking for a probabilistic branching time logic with fairness. Distrib Comput 11(3):125–155 · doi:10.1007/s004460050046
[2] Bertsekas D, Tsitsiklis J (1991) An analysis of stochastic shortest path problems. Math Oper Res 16(3):580–595 · Zbl 0751.90077 · doi:10.1287/moor.16.3.580
[3] Billingsley P (1979) Probability and measure. Wiley, New York · Zbl 0411.60001
[4] Chadha R, Viswanathan M (2010) A counterexample guided abstraction-refinement framework for Markov decision processes. ACM Trans Comput Logic (to appear) · Zbl 1351.68154
[5] Chatterjee K, de Alfaro L, Henzinger T (2004) Trading memory for randomness. In: Proc. 1st int. conf. quantitative evaluation of systems (QEST’04). IEEE Comput. Soc., Los Alamitos, pp 206–217
[6] Chatterjee K, Henzinger T, Jhala R, Majumdar R (2005) Counterexample-guided planning. In: Proc. 21st conference in uncertainty in artificial intelligence (UAI’05), pp 104–111
[7] Cheshire S, Adoba B, Guttman E (2002) Dynamic configuration of IPv4 link-local addresses (draft August 2002). Zeroconf Working Group of the Internet Engineering Task Force ( www.zeroconf.org )
[8] Clarke E, Grumberg O, Jha S, Lu Y, Veith H (2000) Counterexample-guided abstraction refinement. In: Emerson A, Sistla A (eds) Proc. 12th int. conf. computer aided verification (CAV’00). Lecture notes in computer science, vol 1855. Springer, Berlin, pp 154–169 · Zbl 0974.68517
[9] Condon A (1992) The complexity of stochastic games. Inf Comput 96(2):203–224 · Zbl 0756.90103 · doi:10.1016/0890-5401(92)90048-K
[10] Condon A (1993) On algorithms for simple stochastic games. Advances in computational complexity theory. DIMACS Ser Discrete Math Theor Comput Sci 13:51–73 · Zbl 0808.90141
[11] D’Argenio P, Jeannet B, Jensen H, Larsen K (2001) Reachability analysis of probabilistic systems by successive refinements. In: de Alfaro L, Gilmore S (eds) Proc. 1st joint int workshop process algebra and probabilistic methods, performance modelling and verification (PAPM/PROBMIV’01). Lecture notes in computer science, vol 2165. Springer, Berlin, pp 39–56 · Zbl 1007.68131
[12] de Alfaro L (1999) Computing minimum and maximum reachability times in probabilistic systems. In: Baeten J, Mauw S (eds) Proc. 10th int. conf. concurrency theory (CONCUR’99). Lecture notes in computer science, vol 1664. Springer, Berlin, pp 66–81 · Zbl 0949.93082
[13] de Alfaro L (1997) Formal verification of probabilistic systems. Ph.D. thesis, Stanford University
[14] de Alfaro L, Roy P (2007) Magnifying-lens abstraction for Markov decision processes. In: Damm W, Hermanns H (eds) Proc. 19th int. conf. computer aided verification (CAV’07). Lecture notes in computer science, vol 4590. Springer, Berlin, pp 325–338 · Zbl 1135.68486
[15] de Alfaro L, Henzinger T, Kupferman O (1998) Concurrent reachability games. In: Proc. 39th symp. foundations of computer science (FOCS’98). IEEE Comput. Soc., Los Alamitos, pp 564–575 · Zbl 1154.91306
[16] de Alfaro L, Henzinger T, Kupferman O (2007) Concurrent reachability games. Theor Comput Sci 386(3):188–217 · Zbl 1154.91306
[17] Desharnais J, Gupta V, Jagadeesan R, Panangaden P (2003) Approximating labelled Markov processes. Inf Comput 184(1):160–200 · Zbl 1028.68091 · doi:10.1016/S0890-5401(03)00051-8
[18] Fecher H, Leucker M, Wolf V (2006) Don’t know in probabilistic systems. In: Valmari A (ed) Proc. 13th int. spin workshop on model checking of software (SPIN’06). Lecture notes in computer science, vol 3925. Springer, Berlin, pp 71–88 · Zbl 1178.68341
[19] Graf S, Saidi H (1997) Construction of abstract state graphs with PVS. In: Grumberg O (ed) Proc. 9th int. conf. computer aided verification (CAV’97). Lecture notes in computer science, vol 1254. Springer, Berlin, pp 72–83
[20] Han T, Katoen JP, Damman B (2009) Counterexample generation in probabilistic model checking. IEEE Trans Softw Eng 35(2):241–257 · doi:10.1109/TSE.2009.5
[21] Hermanns H, Wachter B, Zhang L (2008) Probabilistic CEGAR. In: Gupta A, Malik S (eds) Proc. 20th int. conf. computer aided verification (CAV’08). Lecture notes in computer science, vol 5123. Springer, Berlin, pp 162–175 · Zbl 1155.68438
[22] Hinton A, Kwiatkowska M, Norman G, Parker D (2006) PRISM: A tool for automatic verification of probabilistic systems. In: Hermanns H, Palsberg J (eds) Proc. 12th int. conf. tools and algorithms for the construction and analysis of systems (TACAS’06). Lecture notes in computer science, vol 3920. Springer, Berlin, pp 441–444
[23] Hurd J, McIver A, Morgan C (2005) Probabilistic guarded commands mechanized in HOL. Theor Comput Sci 346(1):96–112 · Zbl 1080.68063 · doi:10.1016/j.tcs.2005.08.005
[24] Huth M (2004) An abstraction framework for mixed nondeterministic and probabilistic systems. In: Baier C, Haverkort B, Hermanns H, Katoen JP, Siegle M (eds) Validation of stochastic systems. Lecture notes in computer science, vol 2925. Springer, Berlin, pp 419–444 · Zbl 1203.68099
[25] Kattenbelt M, Kwiatkowska M, Norman G, Parker D (2008) Game-based probabilistic predicate abstraction in PRISM. In: Proc. 6th workshop quantitative aspects of programming languages (QAPL’08) · Zbl 1286.68316
[26] Kattenbelt M, Kwiatkowska M, Norman G, Parker D (2009) Abstraction refinement for probabilistic software. In: Jones N, Muller-Olm M (eds) Proc. 10th int. conf. verification, model checking and abstract interpretation (VMCAI’09). Lecture notes in computer science, vol 5403. Springer, Berlin, pp 182–197 · Zbl 1206.68090
[27] Kemeny J, Snell J, Knapp A (1976) Denumerable Markov chains, 2nd edn. Springer, Berlin · Zbl 0348.60090
[28] Kwiatkowska M, Norman G, Parker D (2006) Game-based abstraction for Markov decision processes. In: Proc. 3th int. conf. quantitative evaluation of systems (QEST’06). IEEE Comput. Soc., Los Alamitos, pp 157–166
[29] Kwiatkowska M, Norman G, Parker D, Sproston J (2006) Performance analysis of probabilistic timed automata using digital clocks. Form Methods Syst Des 29:33–78 · Zbl 1105.68063 · doi:10.1007/s10703-006-0005-2
[30] Kwiatkowska M, Norman G, Parker D (2009) Stochastic games for verification of probabilistic timed automata. In: Ouaknine J, Vaandrager F (eds) Proc. 7th international conference on formal modelling and analysis of timed systems (FORMATS’09). Lecture notes in computer science, vol 5813. Springer, Berlin, pp 212–227 · Zbl 1262.68125
[31] Larsen K, Skou A (1991) Bisimulation through probabilistic testing. Inf Comput 94:1–28 · Zbl 0756.68035 · doi:10.1016/0890-5401(91)90030-6
[32] McIver A, Morgan C (2004) Abstraction, refinement and proof for probabilistic systems. Monographs in computer science. Springer, Berlin
[33] Monniaux D (2005) Abstract interpretation of programs as Markov decision processes. Sci Comput Program 58(1–2):179–205 · Zbl 1088.68039 · doi:10.1016/j.scico.2005.02.008
[34] Norman G (2004) Analysing randomized distributed algorithms. In: Baier C, Haverkort B, Hermanns H, Katoen JP, Siegle M (eds) Validation of stochastic systems. Lecture notes in computer science, vol 2925. Springer, Berlin, pp 384–418 · Zbl 1203.68326
[35] PASS tool homepage. http://depend.cs.uni-sb.de/PASS/
[36] Pierro AD, Hankin C, Wiklicky H (2006) Abstract interpretation for worst and average case analysis. In: Reps T, Sagiv M, Bauer J (eds) Program analysis and compilation, theory and practice, essays dedicated to Reinhard Wilhelm on the occasion of his 60th birthday. Lecture notes in computer science, vol 4444. Springer, Berlin, pp 160–174
[37] PRISM web site. http://www.prismmodelchecker.org/
[38] Segala R (1995) Modelling and verification of randomized distributed real time systems. Ph.D. thesis, Massachusetts Institute of Technology
[39] Sen K, Viswanathan M, Agha G (2006) Model-checking Markov chains in the presence of uncertainties. In: Hermanns H, Palsberg J (eds) Proc. 12th int. conf. tools and algorithms for the construction and analysis of systems (TACAS’06). Lecture notes in computer science, vol 3920. Springer, Berlin, pp 394–410 · Zbl 1180.68179
[40] Shapley L (1953) Stochastic games. In: Proc. national academy of science, vol 39, pp 1095–1100 · Zbl 0051.35805
[41] Smith M (2008) Probabilistic abstract interpretation of imperative programs using truncated normal distributions. In: Aldini A, Baier C (eds) Proc. 6th workshop on quantitative aspects of programming languages (QAPL’08). Electronic notes in theoretical computer science, vol 220(3). Elsevier, Dordrecht, pp 43–59 · Zbl 1286.68094
[42] Stoelinga M, Vaandrager F (1999) Root contention in IEEE 1394. In: Katoen JP (ed) Proc. 5th int. AMAST workshop real-time and probabilistic systems (ARTS’99). Lecture notes in computer science, vol 1601. Springer, Berlin, pp 53–74
[43] Wachter B, Zhang L (2010) Best probabilistic transformers. In: Barthe G, Hermenegildo M (eds) Proc. 11th int. conf. verification, model checking and abstract interpretation (VMCAI’10). Lecture notes in computer science, vol 5944. Springer, Berlin, pp 362–379 · Zbl 1273.68244
[44] Wachter B, Zhang L, Hermanns H (2006) Probabilistic model checking modulo theories. In: Proc. 4th int. conf. quantitative evaluation of systems (QEST’07). IEEE Comput. Soc., Los Alamitos, pp 129–138
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.