Towards An Objective and Scientific CFP Methodology

scientistsI, like many in the information security industry, submit talks to a number of conferences every year. The more conferences I submit to, however, the more apparent it becomes (at least to me) that a more scientific approach to the call for papers/proposal (CFP) process is required to reduce bias. I’m not saying that any or all of the CFP committee participants, or the conference itself, is guilty of malicious or intentional bias. Science has shown that there will always be irrational cognitive biases, whether intentional or not, that affect our decision making process. In a CFP process this could manifest itself as bias towards any number of things such as a particular topic, an individual’s past, sexual orientation or identification, company or industry affiliation, and even the grammar of the submission itself.
 
I see the CFP process as a reproducible experiment. As such, an experiment of this nature requires a number of things to be conducted successfully in a measurable and repeatable fashion. The following list of ideas are the result of a personal brainstorm of what I would like to see included as a part of the CFP process (in no particular order):

  1. Selection committee participants should be publicly disclosed before the CFP process opens.
    • This committee should not be altered throughout the course of the selection window.
    • If the committee membership must be altered due to extenuating circumstances the change must be promptly communicated to all submitters.
  2. Selection should be conducted using a round-robin double-blind selection methodology.
    • No reviewer should know the name of the submitter during the course of the review process to protect the objectivity of the reviewer and the integrity of the process.
  3. Selection committee members should immediately report any contact or attempted influence from a submitter to select their submission(s) or to reject the submission(s) of another.
    • The inappropriate conduct must be reported to the program chair(s) or conference organizer(s) for a determination on whether or not the submission shall be removed from consideration.
    • If the selection committee member feels that they have been unduly influenced by a submitter or another committee member, they should have the option to recuse themselves from a decision on this particular submission or speaker.
  4. Any talks submitted with the name of the individual and, in most cases, the name of the submitter’s company should be immediately bounced back to the submitter for revision or removed from contention.
    • This rejection should be performed by an objective reviewer (gatekeeper) that cannot influence the selection committee and also is not involved in the selection process after initial qualification of the submission.
    • Note that I say “most cases” with regards to the naming of the submitter’s company. There will always be cases, for example a talk about scaling an application, that will be enhanced by aligning the talk with a particular company to add context and veracity to the talk.
  5. Selection committee members should be required to provide commentary for every judged submission.
    • This allows for proper feedback to be communicated to the submitter as to why their submission was accepted or rejected and allows the submitter to grow as a presenter.
  6. Selection committee members should be sourced from the attendee profile (e.g. industry, geography, career level, etc.) that the conference is targeting.
    • What’s the point of having a group of committee members that don’t represent the people actually attending the conference?
  7. Selection committee members should be rotated on an annual basis or a predetermined period of time.
    • The committee should also be composed of sixty percent (60%) new members every year for which the event is held.
  8. Selection committee members should serve on no more than three (3) similar selection boards for different conferences.
    • This prevents the incestuous nature of the security community from spilling over into adjacent conferences.
  9. The CFP submission, voting, and selection data should be made available upon request or published on a publicly accessible site and should include:
    • CFP submission details (such as title, abstract, outline, etc.)
    • Committee member voting results for each submission
    • Committee member commentary for each submission
    • Committee member response time for each submission (from receipt to response)
  10. The entire CFP process should be revisited after the conference concludes to allow for feedback and changes to be surfaced for conversation before the next event.
    • This will help the conference organizers refine the process
I would like to say that this is a perfect CFP experiment but the unfortunate reality is that security conferences will always have external influencers such as the requirement for vendor funding in exchange for speaking preference, out-of-process “friends and family” submission acceptance, and the selection of an “industry rockstar” to present in an effort to draw additional paid attendees. These are often necessary evils, or in some cases personal preferences, in the mind of the conference organizers and may be unavoidable. All I can say is proceed with caution. Contrary to popular belief you, as a conference organizer or CFP committee member, are not Steve Jobs and likely do not have the magical foresight to know what an attendee wants before they know they want it.
 
Now for some ideas I’d like to leave you with…
 
Why not try opening a CFP for CFP committee participants?
This may feel like an unnecessary step for a conference to undertake but it may result in hungry up-and-coming security practitioners to get a seat at the table. One alternative might be to hold a contest, such as an essay submission or blog post, to entice new committee members into the fold.
 
What about letting the conference participants/attendees of the conference vote on the talks to be presented?
This lets the people going to the conferences to learn about topics/ideas that truly interest them – and not the conference organizers or CFP committee. I’ve seen this work at several conferences if the voting is gated properly (i.e. limiting voting to an individual’s IP address and/or registration). Plus, if the attendees dislike the program/content/speakers they really have no one to blame but themselves and may adjust their voting at subsequent conferences.
 
Why not mandate a 60% (or higher) new speaker rule?
The current speakers on the “security conference circuit” are getting older and busier. Some of the most prolific, engaging, and entertaining conference speakers do not speak at as many conferences as they used to because their careers, roles, and lives have changed. Individuals such as Chris Hoff, Jeremiah Grossman, Dave Shackleford, Alex Hutton, Rich Mogull, Jack Daniel, and Mike Rothman no longer speak at seemingly every conference like they once did. They, like myself, are not as young as they used to be and are, like myself, extremely busy. It’s only a matter of time before these excellent presenters voluntarily check themselves into the “conference speaker retirement home” and let the younger generation take over.
 
If conferences start pushing for a 60% new speaker makeup we can ensure that the industry continues to have both great content AND seasoned speakers with an ability to communicate said content. The old guard should also be mentoring these new speakers and helping them sidestep all of the issues that they themselves encountered on their respective road to speaker stardom.
 
So what are your thoughts on the matter?
|||
Scroll to top