Skip to main content

The Micro-Processes of Searching Complex Design Landscapes

Ademir-Paolo Vrolijk1, Esdras Paravizo2, Nathan Crilly2, Alison Olechowski1

1University of Toronto, 2University of Cambridge

Presented at the 24th International Conference on Engineering Design – Bordeaux, France

MOTIVATION

Engineers aim to design complex, novel artifacts that perform as well as theory estimates (Kossiakoff et al., 2011). To do so, engineers make a series of choices during the artifact’s design that get them to their goal. Each choice updates and refines the artifact as its real performance is demonstrated against what may (or may not) be possible (Hazelrigg, 1998). Sometimes, a choice also changes the constraints of what engineers are designing towards (Gralla et al., 2016). In short, each choice incrementally solidifies what is and reveals more of what is possible, opening the way for the next one after that (Schon & Wiggins, 1992; Simon, 1973).

Scholars studying problem-solving have previously abstracted this search process as navigating a rugged solution landscape (Levinthal, 1997; Loch et al., 2003). Specifically, the landscape depicts alternative versions of the same artifact (x,y-plane), varying the design choices and their related performance (z-axis). The progress between the first and final design forms a path on this landscape. Based on work comparing the design behaviors of experts and novices, these paths may differ systematically across engineers with different expertise (e.g., Atman et al., 1999; Cross & Cross, 1998).

However, while scholars have theorized many design process models (e.g., Wynn & Clarkson, 2018), few of these models consider how artifact performance estimates drive design decisions. Additionally, scholars have used rugged landscapes to empirically represent a set of solutions to a well-defined problem (e.g., Knudsen & Srikanth, 2014), but their usage concerning complex ones is scarce.

To address that gap, we first focus on representing the search process on a rugged solution landscape. In particular:

RQ: How can we visualize complex design as navigating a rugged landscape?

RESEARCH DESIGN

Observational Study

To examine visualization techniques that can meaningfully depict the search process, we chose a context where:

  • The desired output of the design process was artifact performance (and not, for example, the number of solutions).
  • The sequence of artifact versions and related system performance was observable. We would then map this onto the solution landscape.
  • The search patterns of engineers with “polar” design experience (Eisenhardt & Graebner, 2007) could be characterized; these groups are more likely to exhibit differences in how they search.

Setting

We chose to leverage an ongoing research project as our research setting. In particular, recent work by Paravizo and Crilly (2023) investigates the effects of creative performance feedback on design outcomes. Their setting presented a useful toy problem to explore our preliminary research directions. For additional information regarding this project, follow this link: osf.io/zavxc.

At the core of the toy problem is the game Poly Bridge 2. It simulates the multi-variable problem of designing bridges across various waterways. The game grants the same budget to all players per level. Players then design a bridge that successfully supports a car going across, aiming for low cost and low stress to the bridge’s components. Together, these factors create an appropriate context to pursue our research question.

Additionally, we relied on public solutions to support this work. Poly Bridge 2’s community has made their solutions to many game levels public; their bridge structures vary greatly in their structures and materials used. We drew on the variation of these designs and their resulting performance to craft the solution landscape for our analysis. We provide our three-dimensional solution landscape below.

BRIDGE DATA AND ANALYSIS

Our analysis of the bridge solutions proceeded as follows:

  • We captured all bridge solutions (j) produced by each designer (i)
  • We abstracted each solution into a wireframe diagram, and
  • We characterized each solution according to categorical and numerical variables that describe crucial design choices and their performance. We describe these characterizations in the figure below.

Participant Data and Analysis

For our preliminary analysis, we observed the search behavior of four participants playing one level of Poly Bridge 2. In the table below, we summarize:

  • Their experience with engineering design,
  • The number of solutions they created during their search,
  • The performance of their first and final bridge, representing where they started and ended their search, and
  • Their best-performing bridge and at what point along their search path they achieved it (a normalized measure of a participant’s ith solution).

Note that we normalized these performance scores across the solution landscape: the best bridge scored a 1, and the lowest a 0.

PRELIMINARY DIRECTIONS

Results

Our work produced a series of visualizations depicting the participants’ search. To do so, we mapped the data extracted from the participants’ bridge solutions onto the solution landscape generated by the community. As a first step into understanding how engineers search, we found that these visualizations adequately reflected a complex landscape of solutions. First, the visualizations differentiated between solutions with different design choices: different bridges had different [x,y] coordinates. Likewise, it depicted each solution’s performance: bridges with similar performance values had a similar [z] coordinate. Additionally, plotting one participant’s solutions on the landscape showed which bridge designs they explored and which remained unknown.

We also found that these visualizations adequately reflected a search process. In particular, we observed that each participant had a distinct path across the landscape: starting and ending at different locations and traversing different coordinates to get there. Additionally, we observed behavior commonly associated with constructs such as local and distant search (Katila & Ahuja, 2002) and refinement (Wynn & Eckert, 2017).

We show one search path below. In this figure, we show Participant A’s attempt. They completed ten iterations of the bridge across the same waterway: each changing its structure, deck, and resulting performance score. First, the noticeable differences in their first and final design (pictured in callouts) are reflected in the x,y distance between them on the landscape. Second, Participant A’s sequence of solutions, generally, showed incremental changes resulting in incremental performance improvements. This could suggest that A tried to tweak a working design instead of frequently making exploratory changes.

Future Work

This work on search visualizations over a complex landscape is a first step. While these results are promising, we expect to build on them to address how performance can drive design decisions. First, we will collect additional searches in our Poly Bridge 2 level, drawing from a large, diverse sample of engineering designers (Paravizo & Crilly, 2023). These data will allow us to provisionally characterize the patterns engineers display while searching. We then plan to replicate this study on a Computer-Aided Design platform, where the focus will be the design of a mechanical part. This real-world context will refine our search patterns to apply more broadly. These steps will allow us to create robust insights and ensure their relevance to real-world design.

FUNDING

This work draws upon research supported by the Government of Canada’s New Frontiers in Research Fund (NFRF) (NFRFR-2021-00351)

This work was supported by the Engineering and Physical Sciences Research Council (UK) (RG105266/EP/T517847/1)

AUTHORS

Ademir-Paolo Vrolijk
University of Toronto
Esdras Paravizo
University of Cambridge
Nathan Crilly
University of Cambridge
Alison Olechowski
University of Toronto

REFERENCES

Atman, C. J., Chimka, J. R., Bursic, K. M., & Nachtmann, H. L. (1999). A comparison of freshman and senior engineering design processes. Design Studies, 20(2), 131–152. https://doi.org/10.1016/S0142-694X(98)00031-3

Cross, N., & Cross, A. C. (1998). Expertise in engineering design. Research in Engineering Design, 10(3), 141–149. https://doi.org/10.1007/BF01607156

Eisenhardt, K. M., & Graebner, M. E. (2007). Theory Building From Cases: Opportunities And Challenges. Academy of Management Journal, 50(1), 25–32. https://doi.org/10.5465/amj.2007.24160888

Gralla, E., Goentzel, J., & Fine, C. (2016). Problem Formulation and Solution Mechanisms: A Behavioral Study of Humanitarian Transportation Planning. Production and Operations Management, 25(1), 22–35. https://doi.org/10.1111/poms.12496

Hazelrigg, G. A. (1998). A Framework for Decision-Based Engineering Design. Journal of Mechanical Design, 120(4), 653–658. https://doi.org/10.1115/1.2829328

Katila, R., & Ahuja, G. (2002). Something Old, Something New: A Longitudinal Study of Search Behavior and New Product Introduction. Academy of Management Journal, 45(6), 1183–1194. https://doi.org/10.5465/3069433

Knudsen, T., & Srikanth, K. (2014). Coordinated Exploration: Organizing Joint Search by Multiple Specialists to Overcome Mutual Confusion and Joint Myopia. Administrative Science Quarterly, 59(3), 409–441. https://doi.org/10.1177/0001839214538021

Kossiakoff, A., Sweet, W. N., Seymour, S., & Biemer, S. M. (2011). Systems Engineering Principles and Practice (2 edition). Wiley-Interscience.

Levinthal, D. A. (1997). Adaptation on Rugged Landscapes. Management Science, 43(7), 934–950. https://doi.org/10.1287/mnsc.43.7.934

Loch, C., Mihm, J., & Huchzermeier, A. (2003). Concurrent Engineering and Design Oscillations in Complex Engineering Projects. Concurrent Engineering, 11(3), 187–199. https://doi.org/10.1177/106329303038030

Paravizo, E., & Crilly, N. (2023). The effects of creative performance feedback on design outcomes. https://doi.org/10.17605/OSF.IO/ZAVXC

Schon, D. A., & Wiggins, G. (1992). Kinds of seeing and their functions in designing. Design Studies, 13(2), 135–156. https://doi.org/10.1016/0142-694X(92)90268-F

Simon, H. A. (1973). The Structure of Ill-Structured Problems. In H. A. Simon (Ed.), Models of Discovery: And Other Topics in the Methods of Science (pp. 304–325). Springer Netherlands. https://doi.org/10.1007/978-94-010-9521-1_17

Wynn, D. C., & Clarkson, P. J. (2018). Process models in design and development. Research in Engineering Design, 29(2), 161–202. https://doi.org/10.1007/s00163-017-0262-7

Wynn, D. C., & Eckert, C. M. (2017). Perspectives on iteration in design and development. Research in Engineering Design, 28(2), 153–184. https://doi.org/10.1007/s00163-016-0226-3