Conducting Interviews

Planning the interview

An assessment interview requires careful planning.

Setting up the meetings
Be selective about who sets up the meetings. In some cultures, cooperation improves if a senior leader does this.

Interview format
Plan 3 to 5 interviews of 1.5–2 hours each, scoped per topic. SAMM’s business functions are typically a good starting point for your interview topics.

List of stakeholders
Select relevant stakeholders for each interview. For the Governance business function interview you might need people involved in governance and security champions. For the Verification business function interview you might need business analysts, application architects, developers, QA, and/or project managers. Keep the interview group size small and minimize spectators. People open up more in smaller groups.

Pre-interview briefing
Organize a prep kick-off briefing to let interviewees know the purpose of the interview, the format and length of the sessions, which co-workers you will speak to, and the terms of confidentiality. If interviewees are not familiar with SAMM, provide at least a high-level overview of the model.

Live vs. online
Live sessions are preferable over conference calls. People will likely trust you more if they meet you in person and are more likely to open up.

Interview preparation

Work with two people if possible: an interviewer and a note taker.

Study the organization before the interview if you are not familiar with it. Ask them to provide relevant documentation in advance: organizational policies and standards, process-related documents, artifacts from completed activities, etc.

Write an interview guide with open-ended questions based on the information you need to obtain (SAMM questions in this case). The guide provides the structure of a conversation instead of a long list of questions. See the sample interview guide in the appendix.

Book time after the interview to consolidate your notes.

Interview questions

Focus on actuality
Always ask how things have been going, not how things should be. For instance: “When was the organizational policy document last updated?” instead of “Is the organizational policy updated frequently?”.

Ask open-ended questions
Open-ended questions are ideally not copies of SAMM questions, but are meant to get the interviewee to talk on subjects in which the SAMM questions are likely to be answered. You can mention topics the interviewee doesn’t bring up. For instance: “And how about compliance obligations, are they relevant?”.

Be a detective
Avoid questions about what is right or wrong. Find out what the organizational realities are as someone who is simply curious to understand them. Instead of “Do you use checklists during threat modeling?”, ask “Describe the threat modeling process” and listen to see if checklists are mentioned.

The rate trick
Ask people to rate certain things on a scale of 1 to 10 and then ask why. This is a good strategy to get people talking.

Focus on feelings
Asking people about their feelings rather than just their thoughts can encourage them to open up. For instance: “How do you feel about the added value of your threat modeling process?”.

The interview process

Starting the interview
Ask whether you may record the interview. In our experience, recordings and their transcripts can be extremely helpful during note consolidation. However, in some cultures it works better not to record, as even if people allow it, they may feel less comfortable and be less candid.

Break the ice
Invest some time in the interviewees even if it’s just small talk to lighten the atmosphere. If you do a round of introductions, start yourself and make your introduction a bit personal, to invite others to do the same.

It is not an exam
Be courteous, friendly, respectful and humble. The interview is a collaboration, not an interrogation. You want to assess a situation as it is, so you know how to make it better where possible.

Be supportive
Appreciate that people may feel proud or threatened. Avoid negativity or judgment. Listen and encourage responses with enthusiasm: “I see”, “That makes sense”, “Given your risk profile that is a reasonable strategy”. Paraphrase and ask follow-up questions. Be curious.

Ask for artifacts
Ask for evidence for some of the answers. If you do this early in the session, people will realize you might check their answers and try to be more truthful.

The power of silence
Long silences can help people open up. Don’t interrupt.

Keep a natural conversation flow
Allow the conversation to flow naturally. Some of the questions you prepared may be answered without prompting. Pick your questions based on where the conversation is, not per the topic that was next on your list. Your note-taking colleague can help you track which questions have been answered.

Keep it structured
Give interviewees a sense of structure by providing transitions between major topics at quiet moments: “Now that we have discussed X, let’s move to Y.” Avoid sharing a precise structure at the beginning, because you don’t want to restrict people in discussing topics. Natural conversations go back and forth, and it’s better to let them flow.

Note taking
The note taker notes everything. Use verbatim unless completely sure of interpretation. For online sessions you can leverage online transcription. Keep a back-channel with your note taker (e.g. via Slack) but make sure note taking doesn’t cause distraction.

The end of the interview
Consider ending with “Is there anything you think we forgot to discuss?” Consider asking any sensitive questions after the interview is officially done.

Post-interview validation

Present the preliminary assessment result to the assessed organization to validate the outcome and see if they have any suggestions. To make this process effective, the assessment should contain rationale regarding each decision.

Example interview questions

See Example Interview Questions for a full set of open-ended questions covering all SAMM business functions, ordered to follow the SAMM assessment sheet 2.0.