by krabob

What you need to known About OpenGL ES(2) for Mobile Graphics

This is an article to explain quickly the differences between OpenGL for desktop GPU and OpenGL ES(2) for Mobile GPU. You need some OpenGL or 3D API background to understand what it’s about.

Capture d’écran 2014-04-22 à 21.29.32
Some rendering with OpenGL ES 2:  4 passes and 4 shaders are used here.

All Mobile devices today have something in common: a mobile Graphical Processor Unit (GPU) that can be programmed with OpenGL ES and OpenGL ES2. Every iPhone, iPads from apple include a PowerVR, Raspberry Pi embeds a “VideoCore IV”, Android models use all those, plus a wide range of other GPU like NVidia Tegra, adreno, snapdragon. A lot of models, but only 2 API to program them all at the lowest level:
OpenGL ES (quite the same as OpenGL1.x) and OpenGL ES2 (quite the same as.. OpenGL 3)  … and yes, “ES” means a lot of differences…

Everything in this document is both from OpenGL documentation, specific GPU documentations, and my own experience testing with any kind of models.

No, the OpenGL ES driver will not be corrected by a system update in the future.

If you were used to hope for drivers updates on desktop to correct a bug or manage something differently, you have to know that GLES drivers are done once and for all for a given mobile GPU, quite often stands in ROMS and not in the system, and that basically the system (android,…) does not manage that part. and the GPU builder does not care because they just does no support after release.

 Writing on FBO, and using glViewport() and glClear()  differ *Totally*.

 Framebuffer Objects (FBO) is the official way to create offscreen bitmaps and offscreen rendering, and you can link them to a texture Id.
The bad news is Mobile OGL Drivers only manage one FBO context at a time, and a whole FBO or screen are always  internally “tile managed” by the driver, with possibly “implicit superscalar Buffers”, which means for you:

  1. glBindFramebuffer() must be followed by glViewport() and glClear(), if not it will crash on most Mobile GPU. You cannot “come back” to a FBO and continue drawing.
  2. glClear() will clear the whole FBO, not the rectangle given by glViewport(), unlike classic OGL. You absolutely can’t do one glViewport() on half the screen draw, then glViewport() the other half and draw again: impossible on all ES. Due to this, you should always give the whole rectangle to glViewport().
  3.  (Due to 1 and 2) You must render your offscreens FBOs at the beginning of a frame, and then only begin to draw the screen. A nice idea is to have a list of off-screens to render with delegate functions/methods, so your program will automatically “sort” the order of your drawing needs for a given frame.

 Texture Constants for glTexImage2D() are not the same in ES

There is a very wide list of texture pixel internal formats on classic desktop GL, that describes both the encoding and number of component: this is not re-used at all on  OpenGLES(2), because you must have components and encoding separated. So glTexImage2D() have to be used differently. Read your GPU documentation about it.
To program in a way that is compatible on all OpenGL , both mobile and desktop, I have something like this in my headers: (note gl includes files also differs)

// for OpenGL ES 1/2: textures are not set
// use 16 hight bits to set the format enum(GL_UNSIGNED_BYTE,GL_FLOAT)
// the low 16 bits for the number of component enum:
// watch out, glTexImage2D() isn't used the same way either
typedef enum {
} ePixelFormat;
// Classic desktop OpenGL stuff:
typedef enum {
    epf_Byte_RGBA=GL_RGBA8, .
} ePixelFormat;

and in my inits (for FBO actually):

    // OpenGL ES 1 or 2:
    unsigned int components =(unsigned int)( pixelFormat & 0x0000ffff);
    unsigned int format =(unsigned int) ((pixelFormat>>16)& 0x0000ffff);
    glTexImage2D(GL_TEXTURE_2D, ii, components ,
          pixelWidth, pixelHeight, 0, components, format, NULL);
    // OpenGL 1,2 not ES
        glTexImage2D(GL_TEXTURE_2D, ii, pixelFormat ,  
          pixelWidth, pixelHeight, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);

…Then by using the ePixelFormat enum in my code, I can stay compatible on desktop and mobile.

Note something special about texture size:  Textures and FBO can be any pixel size on OpenGL ES and ES2, with all implementations: no need to test for extensions, NPOT (Non power Of Two size) is mandatory on ES (and for ES2, GL’s extension list will not display NPOT just because it doesn’t have to, but it’s here.)

Last word about texture formats: NVidia Tegra allows to have internal float16 RGBA textures (not float32 or else) and to write them as float16 with a FBO and a pixel shader, so you can do some nice “General purpose GPU” (GPGPU) tricks on it. As far as I know, a lot of other GPU declare float texture format extensions but it’s only the loading format. they all are restricted to 8 bit FBO internally. (tested last time in 2014)

With OpenGLES2 (like OpenGL 3 desktop)  No more glColor function, no more matrix management,…

For both Mobile ES and desktop OpenGL, a “2” or greater version number means it is a Shader-based architecture that needs a vertex Shader and pixel Shader to draw something.
If you want to use the real power of your GPU, you have to use shaders.
And if you want shaders on mobile, you’ll have more code to do than with OpenGL 1.

Basically in OpenGL ES2 and OpenGL3, every GL functions that could be also done with shaders using using “attributes” and “uniform”, were removed.
So every glMatrix functions were removed,  you have to declare your own uniform matrix and apply them your own way in the vertex shader. It implies you have your own translate, rotate, multiply, etc matrix implementations… extra SDK often offer their (NVIDIA Tegra SDK, any open source engine,…) I did my own. It’s always useful to patch 4×4 float matrices in a way or another.

OpenGL ES2 GLSL Shaders are a bit differents compared to classic GLSL.

First some simple rules:
a. Your Mobile GPU is far more powerfull than the ARM FPU: It’s always better to have computation in your Vertex Shader rather than in the CPU Float code. Think about it twice, you can have “quite big vertex shaders”, there is no problems.

b. You can’t read textures in vertex shader under OpenGL ES 2. Arggg I hate that. It actually would have been the coolest thing ever, some (rare) mobile GPU and drivers actually does it, but you want to be compatible.

Then the only real serious thing to know about GLSL for mobile:
You must start every Shader with one of these lines:

 precision lowp float;
 precision mediump float;
 precision highp float;

This precision thing only exists for ES shader and is mandatory.
It allows to choose if the GPU would work with 8b float16 or float32
It is meant to be powerful, because lowp would be enough for copying 8bit textures and highp would allow more nice rendering.
This is the default main precision of your shader, but you can then declare for each variable any lowp, mediump or highp precision. Theorically, it will finetune the compilation of the shader. But complex shading will need highp to really work. Some glitches are sometimes caused by mediump.

The Dreaded Android Context loss and other Video Memory management issues…

Even if modern Mobile hardware got huge memories, Androids and other systems are still widely ” One App Focused at a time” based. Before Android 3, Every OpenGL app had to completely free all textures, FBO and VBO each times the system was paused, and you had to re-init all of these when the App restart. Even the screen-rotation-with-sensor implied to do all of that, and for a game with many textures and FBO, it was a hell, not mentioning you are meant to manage GL in a special thread… that the main thread will actually kill your context *before* sending the message on the other thread. Android “Activity” class has special messages about it.  Hopefully, Android 3 introduced setPreserveEGLContextOnPause() which will just prevent that behavior… in most cases. Because yes: you still have to manage the “destroy and recreate the GL video memory, but you can keep the rest” thing.

and you know what ? it’s great to do that. I have 2 level of creation/ destruction messages in my whole engines , one for video memory and the other for CPU memory. As I can compile for destop and mobile GL, it also resolves the nasty windows and OSX bug in SDL that destroy the context when you change the size of the screen.

That’s all, have a nice code.


by krabob

Mastering multicore: Parallelizing any algorithms… and how my resource library helps

As in any big project, I use several program libraries for Math Touch Book.
I actually made a brand new Graph Theory API for it, to fit my needs. But I use another one I did between 2006 and 2009, to manage what is called application resources. This was a very interesting management library called Veda. In the case of Math Touch Book, I just re-used it to manage icons, screens, and messaging between objects. Thanks to Veda, If I need one more image for a button, I only add the image once and I have it both in the Android and iOS Apps, and for any future implementation. If I want to get rid of unused images or doubles, it is also automatized for all app versions.

Capture d’écran 2016-02-09 à 13.14.30
Managing external ressources with the Veda Lib in Math Touch Book. The interface is created automatically for any type of objects (images, fonts, sounds,…)

Veda is like a C++ patch where you add some declaration to the class you want to be managed. Then you have special serializable members (serializable means loadable/ savable/clonable). with basic data types of any kind, and one of the magic was special intelligent pointers that allowed to view your whole data as a graph with object links and dependencies. It was first made to experiment with “naked object” design pattern, which is about having interfaces to edit objects by just declaring objects, with no additional code. I also had some experiments with Java serialization, and my old xml based Amiga engine “Karate” was lacking automatic interfaces.
For your information, it is opensource and available here (and you even have Sega Dreamcast demos done with it !):

Veda source and binaries… mostly for windows xp

and Tons of Veda docs were “doxygened” here.

The interface creator for Veda in 2008, showcasing some 3D. This was a Microsoft MFC implementation at the time. The current purple interfaces in other screenshots are the current SDL implementation.

As your whole objects could be managed as a graph, It was made possible to have incremental step-by step initializations for it: when You ask the construction phase of an object, It could need another linked object to be created first, and this second object could also be bound to a third ante-previously initialized object.

At the time veda was made, it had implementation for a 3D engine (that was drawn through an abstract engine, with 5 or 6 different implementations !) . In that case, I had a veda Script classes, which needed some 3D world, which needed 3d objects and cameras, which needed another 3D objects and 3D textures, which needed images of any kind.
But anything in computer science could be Veda-managed, not only 3D engines.

At left, the object list sorted by class. A blue rod meant “created”. editing a leaf object could lead to the de-creation of currently unused objects, dependant of it. The graph was used to manage object updates during edition. On the Second line of the edition deck, you can see an “intelligent pointer” to a texture. You could choose to preview an object with that GUI (here the torus) and to edit another one with the editor Deck. So you could finetune a texture, while watching the result on the object.

 All those objects were linked with intelligent pointers, and an algorithm was able to list all the objects in a leaf-to-root order, to assume you created the leaf objects and then only the dependent ones. A leaf object could be pointed many times, and any object could point another (exception was: you could not “ring” pointers), so it was really a Graph, not a tree.
So incremental initialisations were automatized, Interfaces were automatized, links and data were dynamics, whatever may be the size of the whole graph . My previous space video game “Earths of alioth” also used Veda to manage resources, and its nice progress bar at start, is done thanks to Veda.IMG_1207

At the time, I couldn’t cease to discover new advantages of Graph-theory managed resources in veda, that explains the never ending list of its features.

One of the biggest ideas came with the advent of multi core CPUs...
Since the mid-2000’s, we have efficient multi-core CPU, which means 2 or more programs can run at the same time, not only because of Preemption (which means: multi-tasking with one CPU), but because you really have multiple CPU working, which was impossible before the 2000’s, because of data cache and hardware memory issues: 2 CPU accessing the same memory would trash the memory.

Capture d’écran 2016-02-09 à 12.43.20
level design in “Earths Of Alioth” was edited with an automatized Veda interface !

But then, the problem was to create multi-core programming. in most cases, programmers didn’t change their habits: you use multi-core with long-known threads, and usually, a thread works on its own job, with its own data. For example, a thread would mix sounds alone, when a main thread would do something else.
In the case of very large computation to be performed using the same data, it is often hard to split an algorithm to make the CPU work in parallel. Because you have to be sure the data you read is coherent with what the other CPU are doing in the same time.

Basically, such an API like Veda could do that: for example, you can delegate the inits of whole branches of objects to other CPU threads when you know they are on a “separated branch”, and recover those objects to another thread when initialized.

To make it short and generalize it, knowing the graph of dependencies between objects, makes it possible to automatise the way multiple CPU can share their work with it: it’s just about some lines of Graph-theory algorithms to do that.

Thanks for your attention and have a nice day !



by krabob

Post Media and the end of Static Culture in Digital Short Films.

  Years ago I was frequently terrifying my rare friends in bars with completely improvised and bizarre artistic manifestos. Here is a brand new one:

Come as you are. is originally a group of science students, artists and enthusiasts,   that has been involved in real-time experimental short film on computers since the beginning of the nineties, at a time a lot of things were happening in the underground computer scene through Europe.

Capture d’écran 2016-01-31 à 17.36.10 Early before 1990, Hackers started to explore experimental real-time effects on computers, with graphics and musics. This movement, rapidly spread, involved a lot of research, both technical and artistic.

Later then, there was a paradox you could read between the lines of those productions in the nineties, about nothing but the aim of these numerous short films: Capture d’écran 2016-01-31 à 17.36.46
Was it Technical or Artistic ? Was it Technical demonstrations through Art or Art through
Technical demonstrations ? Or both ? Should Short computer experimental films be made for everyone, or for a finite number of amateurs ?

There were occasional discussions about it, that did not prevent the movement to evolve and follow technical advances. On the Art side, it is noticeable that a lot of different shapes, references to one or another culture, sequencCapture d’écran 2016-01-31 à 17.56.33e montage techniques, experimentations with sound and image symbiosis, exploded during the nineties in these movies, like nowhere else.


Then came the word.

Aside that, Industrial computer science, and marketing, were also going their own way during that period. The advent of the CD Rom, around 1994, was sold with a new Word, that would solely define the links to come between Art and Computer Science: Multimedia.multimerdia
Basically, your computer would be “Multimedia” because it was not anymore that sparkling grey brick lost in a corner of your room. You could have images and sounds now. It was officially allowed to do Art on Computers, thanks for that word.
The problem was, a lot of people were actually doing exactly that since 10 years, and yet that Art was not recognized as valuable, except rare exceptions. And the word didn’t help.

“Hello, I want to buy a Multimedia Computer.”

Capture d’écran 2016-01-31 à 17.37.45You could read the word “Multimedia” during years on ads and brochure, until the mid 2000’s: the fashion more or less lasted a decade. At that time, kids were already raised with internet, and the adults had more frequently computer knowledge and computer culture, not mentioning all media becoming digital.
This, with the advent of video servers being the final nail in the coffin, changed completely the way people used to see computer-for-computer Art, by flattening every symbols, every cultures, to the same level. Whatever you were doing, music with spoons, knitting bags, reviving forgotten musical genres, or Programming effects on a 30 year old 8 bit computers, all these suddenly became OK and could take recognitions.Capture d’écran 2016-01-31 à 17.40.23

The story of Pixel Art is very relevant of this:
Pixel artists used to live by their work in the eighties, drawing characters for video games. It was a difficult art, involving a large set of techniques, they had to be graphic designer, animators, and they had to deal with technical constraints in size, colors, and memory weight. The few people actually able to do that were then praised by their employers.
It all changed in the middle of the nineties. People wanted 3D games and Pixel Art became obsolete quite suddenly. For a long time. Until the Post-Media era.

Paradoxically, labelling a culture leads to restricting that culture. A dead culture is not the one you haven’t heard about, but more likely the one no-one practices, the one that does not take elements from other cultures.

“Hello, I want to buy a Post Media Computer”

Recently the Art world, Gallery Owners and Critics, for the first time in history, arrived at the sampostmediabooke analysis about culture and state of the Art as could do an independent underground demomaker, and even more, using the same words: Post Media. Signs of the time: pixel Artists like eBoy are nowadays praised in galleries. Displaying a background of underground computer geek and experimenter does not scare anymore, quite the reverse, it opens doors.
So what art can you do in an environment where everything is possible, where everything is open ? Isn’t it the time to deeply reshape the Frames of art, mix shapes, colors, sounds, in ways they were never shaped before ? Isn’t it the time to really flatten the field of possibilities before your eyes ?
I think so, and I invite you to do so.

  •  The old stuff from m4nkind is visible on this very site:
  • references more than 62000 Computer Realtime Digital Short Films made between 1980 and today in the world.
  • permanently broadcasts democene videos since 2005.