SIGGRAPH and its sibling conferences have lost their way.
Search, and you will find many meta-articles on how to get your work accepted by SIGGRAPH. From reviewers you will read about a capricious notion called "SIGGRAPH quality" which includes properties akin to completeness and polish. Reviewers pat themselves on the back for maintaining a high standard of quality. But it amounts to awkward, ambiguous and counter-productive elitism.
Computer science adopted the same recipe as more mature hard science disciplines such as physics: introduction, methods, results, conclusion.
Even if this recipe works for peer-reviewed journal articles, it fails for conference talks. Because conferences are where birds of a feather confer to discuss nascent ideas, not to present only polished ideas.
(But also read Shewchuck's "Three Sins of Authors in Computer Science and Math", a constructive rant about mindlessly following the de facto standard format of computer science papers.)
The SIGGRAPH submission process is broken: Reviewers judge submissions and they accept or reject without a rebuttal cycle. This approach has numerous problems: too fast, too shallow and too strict.
Time pressure is so intense that editors send papers to whoever is available, including non-experts. Reviewers feel obliged to participate so instead of declining, they write uninformed opinions. To absolve themselves of their charlatanism, they self-judge their expertise on the subject; reviewers can grade themselves as having little knowledge of the paper's topic. Self-evaluation of expertise is literally part of the review form. Instead of rejecting reviews from non-experts, that grade simply becomes part of the review score as though it has any quantitative value whatsoever. That should be a red flag that their process has issues.
Then comes the kicker: If the article seems incomplete or incorrect, it is rejected without cyclic rebuttal. Yes, authors can rebut, but unlike with a legitimate journal the rebuttal is a one-way missive and the submission process more like fire-and-forget than the cyclic refinement process that should occur for a journal. If an idea has value, then authors and reviewers should work together to distill and communicate that idea.
But why? SIGGRAPH justifies this bizarre process because, they claim, it provides a forum to publish ideas faster than alternatives, where the refinement process can introduce delays. It's not uncommon for an article to take a year from submission to publication, whereas in SIGGRAPH, it takes merely months.
Fine, except that each conference only occurs once a year anyway. So you only win if your idea and its work phase-aligned with the conference where you choose to present it.
But, you protest, there are multiple conferences, distributed throughout the year. Indeed, but in computer science, you are only allowed to submit your paper to a single conference. Say conference A takes submissions in January, and conference A occurs in June. Conference B takes submissions in April and occurs in September. If you submit to conference A, you can't submit to conference B because it would count as "multiple submission" and will be rejected from both. So in practice, rejected papers take longer to reach their audience than if they stuck with journals. So the rapidity argument is specious.
But all this is beside the point. Are conferences the right venue for presenting polished ideas? No, not at all. They're places where ideas go to bake, where you present your works-in-progress specifically so you can engage your peers in the cooperative process of refinement. At conferences, you seek collaborators.
That's the idea anyway. But academic cooperation is a myth. In the battle for tenure, academics compete more intensely than those in industry. Oh, sure, industry has its trade secrets and patents and a very few high-profile ephemeral products that briefly have super-secret development phases. But mostly, employees migrate between companies, and everybody takes pleasure in collaborating with everybody in their industry.
One of the best talks at GDC this year was entitled "Frames, Sparsity and Global Illumination: New Math for Games", by Robin Green from Microsoft. It presented an alternative to spherical harmonics as a basis for representing a light field distribution. The approach seems viable and likely an improvement to SH. But that's not why it was a good talk. It was good because it was incomplete. We come to conferences to see the cutting edge, not to see the canon. We want to know what's coming, not what's been done. We want to participate, not spectate. And we can participate more when ideas have not yet crystalized.
Mind you, GDC has many flaws. Their selection process clearly focuses on whether the authors have associated recent, big-name titles, or whether they are luminaries who will draw a big crowd. But GDC is a for-profit event, so you can't blame them for that.
But SIGGRAPH, I3D, CHI, Eurographics and HPG oranizers should know better, and have confidence that incomplete ideas will not mire reputations -- neither authors' nor conferences'. On the contrary: Conferences should consist of an N-way conversation, where everybody presents something, polished or (ideally) not.
That's what physical sciences have done for centuries. It works wonders.
Yes, SIGGRAPH has its "sketch" sessions but they're underpopulated and overlooked. In contrast, a typical EGU general assembly is mostly composed of what SIGGRAPH calls sketches and lacks what SIGGRAPH calls "paper talks". The 2011 EGU GA had over ten thousand participants and over twelve thousand presentations. The number of presentations exceeded the number of participants! Now, that's how it's done: on average each participant provided more than one presentation. In dismal contrast, SIGGRAPH 2011 had twenty-five thousand attendees (can't call them participants) and merely a few hundred presentations. A tiny fraction of attendees present anything at all. The rest spectate. Not from a lack of material; they were actively excluded from communicating their ideas.
Reserve cooked ideas for proper journal publications, monographs and textbooks. But encourage everybody to participate in conferences by sharing their half-baked ideas.
Subscribe to:
Post Comments (Atom)
2 comments:
Fortunately, this is not true for the modern SIGGRAPH technical papers review process. In that track, papers are reviewed only by experts (and quite carefully). There are five experts on each paper and then the paper and reviews are discussed by a committee of experts. There is a rebuttal phase, and rejected papers can be carried forward with the same reviewers to SIGGRAPH Asia, Transactions on Graphics, and the next year's SIGGRAPH.
I agree that the SIGGRAPH technical paper review process currently errs on the side of conservative, polished, and often esoteric papers over important new ideas that expand the field. For me, that is a downside and I prefer HPG, EGSR, and I3D for new ideas and JGT for practical results. However, I think that the SIGGRAPH process does guarantee quality and that very few incorrect papers are published there. One can certainly submit to and read other venues--SIGGRAPH is not censoring graphics research, just putting a spin on a slice of it.
Unfortunately SIGGRAPH's improvements to the publication process miss the point; SIGGRAPH is a conference, not a journal.
SIGGRAPH tries to be both and fails at both. Furthermore, that community already has journals, so SIGGRAPH should focus on being a conference, where people confer and exchange incomplete ideas and works in progress.
Furthermore, SIGGRAPH's scope has expanded to include topics outside its core competency -- graphics. For example, fluids.
Two examples of erroneous judgement in fluid publications, showing lack of expertise in their judges, come to mind:
- Filament-based smoke with vortex shedding and variational reconnection. This paper contained fundamental mathematical and physical errors in how it treated rotating bodies in fluid: The paper fundamentally assumes that vortex lines are always loops; that is the entire basis of their approach. But vortex lines terminate no rotating bodies.
- A judge repremanded an author for inventing a term. But that term had appeared prominently in the literature for decades.
Both examples lie in my area of study, which is vortex dynamics. So if the examples sound cherry-picked, that's due to the selection bias for where I can readily identify errors. But it also demonstrates that SIGGRAPH judges can be wrong in both directions -- overly permissive and overly critical, especially in an area where it overextends their charter of computer graphics.
How many more papers are erroneously accepted or rejected? It doesn't matter; the point is that they shouldn't concoct inappropriate notions of "SIGGRAPH quality". There should be no such thing. To that extent, quantifying the degree of their judgement error misses the point, which is that conferences are meant to be places where people exchange nascent ideas.
I wish the computer graphics community had more of the attitude toward communication found among physical scientists, where I found a more collaborative spirit.
Thanks for providing your perspective and your strategy for presenting your work, though.
Post a Comment