Me too. I thought about it a bunch when Abrash and Carmack were discussing VR early this year, and while thinking about 48fps movies. Really, I've wanted higher refresh rates ever since the switch to LCDs, and VFR displays would make smooth, high-refresh video much easier.
Since it's unlikely I'll ever be able to get into VFR display tech professionally, for the benefit of anyone working in the industry, here are my thoughts on the subject from February (focused more on video and movies):
Feb. 21, 2013
-------------
INTRO:
In the modern digital age of LCD, LED, plasma, and DLP displays, there's not
much need for refresh rates to be all that high, or even constant. In the days
of CRTs it was necessary to have a constantly refreshing signal coming from the
video source in order to drive the electron beam across the screen and refresh
the image in the phosphors without visible flicker. With film, it was much
easier to run projectors and cameras at a constant rate due to their mechanical
nature (and probable lack of a standard for indicating when to switch frame
rates).
PROPOSAL:
I propose the development of a complete variable frame rate video chain, from
camera, through production, encoding and distribution, and an HDMI/DVI-like
video interface, to display devices. My primary focus of thought thus far has
been on the video interface and display devices.
APPLICATIONS:
Cinema: Recently, The Hobbit was released at 24fps in some theaters, 48fps in
others. As more directors want to experiment with 48fps cinema, why not remove
the restriction to a fixed frame rate entirely? Initially, with widespread
device support, directors and cinematographers could switch between 24fps and
48fps on a scene by scene basis. Eventually though, why not allow the frame
rate to be varied continuously on a per-frame basis? Special effect sequences
could be presented at 120fps, with either a gradual or an abrupt transition
down to 24fps or even lower as desired for emotional or psychological effect.
Variable frame rate displays would also allow mixing of NTSC, PAL, and film
content without the delay of switching video signal refresh rates or using
nasty time stretching or pulldown techniques. Video cameras and encoding
systems could be developed that automatically adjust frame rate based on the
amount of movement detected (to some extent this already exists in video
codecs).
Gaming: Video games would also benefit from variable frame rate displays. No
longer would gamers and game developers have to choose between a tear-free but
delayed game using vertical sync, and a low-latency experience at high frame
rate but with tearing. In a VFR display, the frame would be sent to the
display when the game is ready, rather than the other way around. The content
should be in control, not the device. This way a game could still remain
artifact free if the frame rate drops, without the lag induced by waiting for
the vertical blanking region before swapping buffers.
Mobile: Finally, variable frame rate devices could use much less power for
signal processing and transmission; this would be especially desirable in
battery-powered devices. A tablet running a word processor on an external
display only needs to send an update to the screen once or twice per second to
flash the cursor, saving the power that would be used for reading from video
RAM and driving the display interface.
AREAS OF STUDY:
A very incomplete list of some of the considerations that must be made when
developing a VFR technology suite follows:
The clock rate of an individual frame on the video display interface must be
determined. Should devices negotiate an optimum clock rate upon connection and
use that rate for all frames transmitted? Or should they adjust their pixel
clock rate based on the current frame rate? Using the maximum possible pixel
clock supported by the devices and the copper (or fiber or RF spectra)
connecting them would reduce transmission-induced latency but might increase
power consumption slightly.
A signaling method would need to be devised. Should the display interface
protocol be verbose, with the video source announcing frame rates or frame times
in advance to the display device? Or should the video card just start scanning
out pixels whenever it wants to, and the display just has to deal with it?
Buffering techniques in the display would need to be considered. Should the LCD
(or other) panel be updated as the pixels come in, or should a full frame be
buffered first? How quickly could a buffered frame be shifted into the display
panel? Given the fact that gamers can adapt to and extract greater temporal
information from a higher framerate signal with tearing on a fixed rate display,
there may be some benefit to scanning out the rows of an image as they arrive
(the top row would be displayed a full 16ms sooner than it would otherwise).
Software would need a method of informing the video card that it's done drawing
a frame. To a large extent this already exists in the form of the buffer swap
call, but in a variable refresh rate system, there would be less need for double
buffering to prevent artifacts. The application could draw to the scanout
framebuffer as it pleases, tell the video card to send a scan to the display,
then wait for the video card to notify it that the scan is done. Double
buffering would still be used in games and applications that don't want to wait
for one frame to finish scanning before drawing the next.
FURTHER THOUGHTS:
It would be interesting to go beyond VFR and create a display standard that can
update selected regions of the screen (AKA dirty rectangles). For example, a
video card could send a hypothetical "Start Scan" packet to the display device
that contains the location and size of the region being updated, then stream raw
pixel data that the display device fills into the updated region. For that
matter, the updated region needn't be rectangular.
What about variable resolution display updates as well? This seems to make less
sense in discrete pixel digital displays, but might find application when
displaying low-DPI content in a subset area of a high-DPI/"retina" display.
It seems that a VFR display chain is a lot like double or triple buffering was
back in the CRT days, but one of the buffers has been moved into the display
device itself.
Since it's unlikely I'll ever be able to get into VFR display tech professionally, for the benefit of anyone working in the industry, here are my thoughts on the subject from February (focused more on video and movies):
INTRO:In the modern digital age of LCD, LED, plasma, and DLP displays, there's not much need for refresh rates to be all that high, or even constant. In the days of CRTs it was necessary to have a constantly refreshing signal coming from the video source in order to drive the electron beam across the screen and refresh the image in the phosphors without visible flicker. With film, it was much easier to run projectors and cameras at a constant rate due to their mechanical nature (and probable lack of a standard for indicating when to switch frame rates).
PROPOSAL:
I propose the development of a complete variable frame rate video chain, from camera, through production, encoding and distribution, and an HDMI/DVI-like video interface, to display devices. My primary focus of thought thus far has been on the video interface and display devices.
APPLICATIONS:
Cinema: Recently, The Hobbit was released at 24fps in some theaters, 48fps in others. As more directors want to experiment with 48fps cinema, why not remove the restriction to a fixed frame rate entirely? Initially, with widespread device support, directors and cinematographers could switch between 24fps and 48fps on a scene by scene basis. Eventually though, why not allow the frame rate to be varied continuously on a per-frame basis? Special effect sequences could be presented at 120fps, with either a gradual or an abrupt transition down to 24fps or even lower as desired for emotional or psychological effect. Variable frame rate displays would also allow mixing of NTSC, PAL, and film content without the delay of switching video signal refresh rates or using nasty time stretching or pulldown techniques. Video cameras and encoding systems could be developed that automatically adjust frame rate based on the amount of movement detected (to some extent this already exists in video codecs).
Gaming: Video games would also benefit from variable frame rate displays. No longer would gamers and game developers have to choose between a tear-free but delayed game using vertical sync, and a low-latency experience at high frame rate but with tearing. In a VFR display, the frame would be sent to the display when the game is ready, rather than the other way around. The content should be in control, not the device. This way a game could still remain artifact free if the frame rate drops, without the lag induced by waiting for the vertical blanking region before swapping buffers.
Mobile: Finally, variable frame rate devices could use much less power for signal processing and transmission; this would be especially desirable in battery-powered devices. A tablet running a word processor on an external display only needs to send an update to the screen once or twice per second to flash the cursor, saving the power that would be used for reading from video RAM and driving the display interface.
AREAS OF STUDY:
A very incomplete list of some of the considerations that must be made when developing a VFR technology suite follows:
The clock rate of an individual frame on the video display interface must be determined. Should devices negotiate an optimum clock rate upon connection and use that rate for all frames transmitted? Or should they adjust their pixel clock rate based on the current frame rate? Using the maximum possible pixel clock supported by the devices and the copper (or fiber or RF spectra) connecting them would reduce transmission-induced latency but might increase power consumption slightly.
A signaling method would need to be devised. Should the display interface protocol be verbose, with the video source announcing frame rates or frame times in advance to the display device? Or should the video card just start scanning out pixels whenever it wants to, and the display just has to deal with it?
Buffering techniques in the display would need to be considered. Should the LCD (or other) panel be updated as the pixels come in, or should a full frame be buffered first? How quickly could a buffered frame be shifted into the display panel? Given the fact that gamers can adapt to and extract greater temporal information from a higher framerate signal with tearing on a fixed rate display, there may be some benefit to scanning out the rows of an image as they arrive (the top row would be displayed a full 16ms sooner than it would otherwise).
Software would need a method of informing the video card that it's done drawing a frame. To a large extent this already exists in the form of the buffer swap call, but in a variable refresh rate system, there would be less need for double buffering to prevent artifacts. The application could draw to the scanout framebuffer as it pleases, tell the video card to send a scan to the display, then wait for the video card to notify it that the scan is done. Double buffering would still be used in games and applications that don't want to wait for one frame to finish scanning before drawing the next.
FURTHER THOUGHTS:
It would be interesting to go beyond VFR and create a display standard that can update selected regions of the screen (AKA dirty rectangles). For example, a video card could send a hypothetical "Start Scan" packet to the display device that contains the location and size of the region being updated, then stream raw pixel data that the display device fills into the updated region. For that matter, the updated region needn't be rectangular.
What about variable resolution display updates as well? This seems to make less sense in discrete pixel digital displays, but might find application when displaying low-DPI content in a subset area of a high-DPI/"retina" display.
It seems that a VFR display chain is a lot like double or triple buffering was back in the CRT days, but one of the buffers has been moved into the display device itself.