Rendering with Filament
1. Setup
- need same count for v, vt, n, tangents to allow for per face uvs
- if not, you have to use polygon soup
- use eigen::map to interface with raw buffers
- image format needs to be consistent
- asynchronous buffer copies require you to declare cleanup callbacks
- index buffer size is index count
- make sure offsets and sizes of buffers match what you tell fialment in the vertex buffer attributes
- material pipeline:
- define .mat file
- use matc to compile .filamat file
- load package
- can do this programmatically as well
- filament uses resgen to embed these into the app
- Lit model with normals and tangents
- MeshAssImp calls into assimp and always creates valid values for TANGENTS which allwos for tangent mapping. Is just using noramls possible? Not really. VertexBuffer::populateTangentQuaternions defines a method taht can taek your vertex normals and convert them into the TANGENTS that filament can use.
- Using Eigen and Mikkt space to generate our tangents, bitangents and resulting Quaternion
- What happens when you use a lit model without tangents?
- In a lit model you need to add a light to see anything, duh!
- fully transparent material with spec, see bakedTextureLitTransparent.mat for the cornea - don’t even need a texture here.
- Use mateiral instances instead of creating materials every frame
- culling will remove the geomtery when bounding boxes intersect with viewport. Turn off culling for those render elements that are intersecting with viewport.
- shared textures? body, hip, etc. all share the same texutre. Just need to set the same texture in a different material instance.
- readpixels is async - adding it almost added no over to frame rate. most of the work is done in computing normals and copying memory from cpu
- filament only supports 4 morph targets per renderable, but we have many more htan that, almost 120+ morph targets. Seems ike filament does this by mapping vertex buffers, where as another approach is to encode morphs into a texture buffer. This might be an intersting project to od in its own right and submit back to filament; in the meantime we’re stuck with cpu evaluation of morphs. This github issue seems to indicate that one can write your own vertex shader to do the blendshape evaluation on the gpu. This might be an interesting route instead of modifying the engine itself.
- hardware skins are supported though.
- each vertex has BONEINDICES and BONEWEIGHTS, 4 of each. BONEINDICES are usually USHORT4 and BONEWEIGHTS are FLOAT4. INDICES are local indices into the skin’s transform.
- Each renderable entity has a “skin” which contains an array of transforms. BONEINDICES for each vertex are indices into this array of transforms.
- Skin transforms are:
inverse global transform of node * global transform of joint * inverse bind matrix of joint
- Each render frame must update just the skin transforms for all “skinned” renderables.
- Provide an interface to set the bones as tranforms or Bone object, which is just quaternion and translation. Our skins apparently have non-uniform scaling, so had to send matrices.
- internally filament associates Entity types with different ’Instances’. For example if you call RenderableManager.getInstance(entity) you get back ’renderable’. If you call getInstance on a TransformManager, you get back a ’transform’ instance. These instances are hten used in the filament api to set/get/modify attributes on those instances. Transform instances are always added to entities when “built” using RenderManager::Builder, thus every ’renderable’ instance has a ’transform’ instance. An instance is really just a wrapper around an index. Internally filament keeps a map of entities -> instances.
- If you destroy an entity calling mEngine->destroy(entity), you must also remove it from mScene->remove(entity). You don’t technically have to since filament internally will check if the entities are “alive”, but its porbably not good ot have stale dead entities floating around in the scene.
- using sharedptr destrcutor as a way to signal to app when it should cleanup buffers, etc. in filament when dynamically adding/removing geometries.
2. Materials
- setting specular anti-aliasing is very important if you want to get highlights on your low-roughness materials. Had to set this on a transparent film on cornea geometry to get the highlight on the eye.
3. Optimizations
- compressed textures for higher performance:
https://metalbyexample.com/compressed-textures/
- filament seems to support ETC, ETC2 and ASTC
4. Android Headless EGL
- Look at PlatformEGL.cpp
- The createSwapChain() method returns a created surface, which is the swapchain on android EGL drivers. Instead of using eglCreateWindowSurface() we need to call eglCreatePbufferSurface().
- We can subclass from PlatformEGL and override this method probably to do so.
- seems that version 1.4.3 supports creating swapchains in a headless context. the implementation looks like it should work for macOs and android, but i haven’t had any luck getting ti to work on macos. it simply fails to create a swapchain, even though internally it should just create an NSView.
- NSView on macos doesn’t work, had to sue glfw with a hidden window. Offscreen works just fine on android device with egl, which internally creates a pbuffer.
5. Building on Android on MacOS
It doesn’t seem to work out of hte box, i had to do the following to compile the android NDK version of the filament libraries.
- first you need to build the macos version. You can simply do something like ./build.sh release in teh root dir. This should create a build in out/cmake-release/
- Now in the out/ directory, create a new dir for your android build: mkdir android-release
- use this cmake incantation:
cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE=~/Library/Android/sdk/ndk-bundle/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX
../android-release/filament ../.. = - Make sure you have the right path to the android sdk platform toolchain
- I also had to open up CMakeCache.txt and add teh following entry: EGL:BOOL=ON
- It should now be able to build libfilament.a using `ninja libfilament.a`
- To build `libfilamat.a` you will need to copy the glslminifier application into {CMAKESOURCEDIR}/shaders/. You can find a copy of the glslminifier in the cmake-release/tools/glslminifier directory.
- You might also need to addd glslminifier in your path.
- You should now be able to build everything
6. Render to Texture
7. Vertex Pulling / GPU Blendshapes
8. Building as shared lib
Filament by default builds things as static libraries, which is fine if you are using the same build platform as filament, but when linking to other compilers, e.g., gcc or even with older versions of visual studio, its easier to have filament compiled as dynamic library that is linked in at runtime.
8.1. MacOS
clang++ -shared -fpic test.cpp -std=c++14 -stdlib=libc++ -force_load filament/libfilament.a libs/utils/libutils.a filament/backend/libbackend.a libs/bluegl/libbluegl.a libs/bluevk/libbluevk.a libs/math/libmath.a libs/filaflat/libfilaflat.a libs/ibl/libibl.a -force_load libs/image/libimage.a libs/geometry/libgeometry.a libs/filabridge/libfilabridge.a third_party/smol-v/tnt/libsmol-v.a -framework Foundation -framework Metal -framework OpenGL -framework CoreVideo -framework Cocoa -o libfilament-shared.dylib
Note that we provide a dummy, empty, cpp file to make sure clang is happy.
We do this for filamat as well:
clang++ -shared -fpic test.cpp -std=c++14 -stdlib=libc++ -Wl,-force_load libs/filamat/libfilamat.a libs/utils/libutils.a libs/filabridge/libfilabridge.a shaders/libshaders.a third_party/smol-v/tnt/libsmol-v.a third_party/spirv-cross/tnt/libspirv-cross-core.a third_party/spirv-cross/tnt/libspirv-cross-glsl.a third_party/spirv-cross/tnt/libspirv-cross-msl.a third_party/spirv-tools/source/libSPIRV-Tools.a third_party/spirv-tools/source/opt/libSPIRV-Tools-opt.a third_party/spirv-tools/source/link/libSPIRV-Tools-link.a third_party/spirv-tools/source/reduce/libSPIRV-Tools-reduce.a third_party/glslang/tnt/OGLCompilersDLL/libOGLCompiler.a third_party/glslang/tnt/SPIRV/libSPIRV.a third_party/glslang/tnt/SPIRV/libSPVRemapper.a third_party/glslang/tnt/glslang/libglslang.a third_party/glslang/tnt/glslang/OSDependent/Unix/libOSDependent.a -o libfilamat-shared.dylib
Linking against filament produces some missing symbols in the missing dylib:
Undefined symbols for architecture x86_64: "image::KtxBundle::getSphericalHarmonics(filament::math::details::TVec3<float>*)", referenced from: IBL::loadFromKtx(image::KtxBundle*, image::KtxBundle*) in IBL.cc.o "image::KtxBundle::KtxBundle(unsigned char const*, unsigned int)", referenced from: FilamentWidget::init_impl(void*) in filament_widget.cc.o "image::KtxBundle::~KtxBundle()", referenced from: image::ktx::createTexture(filament::Engine*, image::KtxBundle*, bool)::'lambda'(void*)::operator()(void*) const in IBL.cc.o "filament::rigidTransform(filament::Box const&, filament::math::details::TMat44<float> const&)", referenced from: FilamentWidget::centerCamera() in filament_widget.cc.o FilamentWidget::rayRenderableIntersection(filament::math::details::TVec3<float> const&, filament::math::details::TVec3<float> const&, utils::Entity, float&, float&) in filament_widget.cc.o "image::KtxBundle::getBlob(image::KtxBlobIndex, unsigned char**, unsigned int*) const", referenced from: image::ktx::createTexture(filament::Engine*, image::KtxBundle const&, bool, void (*)(void*), void*) in IBL.cc.o ld: symbol(s) not found for architecture x86_64
Turns out that the KtxBundle class defined in libs/image does not have public visibility parameter set. This diff properly sets the visibility.
--- a/libs/image/include/image/KtxBundle.h +++ b/libs/image/include/image/KtxBundle.h @@ -18,13 +18,14 @@ #define IMAGE_KTXBUNDLE_H #include <math/vec3.h> +#include <utils/compiler.h> #include <cstdint> #include <memory> namespace image { -struct KtxInfo { +struct UTILS_PUBLIC KtxInfo { uint32_t endianness; uint32_t glType; uint32_t glTypeSize;
In addition, prefix the image library with `-forceload` option in clang.
I also get missing symbols during link for rigidTransform:
Undefined symbols for architecture x86_64: "filament::rigidTransform(filament::Box const&, filament::math::details::TMat44<float> const&)", referenced from: FilamentWidget::centerCamera() in filament_widget.cc.o FilamentWidget::rayRenderableIntersection(filament::math::details::TVec3<float> const&, filament::math::details::TVec3<float> const&, utils::Entity, float&, float&) in filament_widget.cc.o ld: symbol(s) not found for architecture x86_64
This diff exposes these symbols properly:
--- a/filament/src/Box.cpp +++ b/filament/src/Box.cpp @@ -20,11 +20,13 @@ using namespace filament::math; namespace filament { +UTILS_PUBLIC Box rigidTransform(Box const& UTILS_RESTRICT box, const mat4f& UTILS_RESTRICT m) noexcept { const mat3f u(m.upperLeft()); return { u * box.center + m[3].xyz, abs(u) * box.halfExtent }; } +UTILS_PUBLIC Box rigidTransform(Box const& UTILS_RESTRICT box, const mat3f& UTILS_RESTRICT u) noexcept { return { u * box.center, abs(u) * box.halfExtent }; }
8.2. Windows
open dev command prompt in vs studio 2019 under Tools menu
Run 64bit cmdline tools in your dev command promopt: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat
cl.exe /DUSRDLL /DWINDLL test.cpp /MT /link /DLL /OUT:helowrld.dll
cl.exe /D_USRDLL /D_WINDLL test.cpp /MT /link /WHOLEARCHIVE:filament.lib geometry.lib /WHOLEARCHIVE:image.lib filabridge.lib utils.lib ibl.lib filaflat.lib backend.lib bluegl.lib opengl32.lib user32.lib gdi32.lib /DLL /OUT:filament-shared.dll cl.exe /D_USRDLL /D_WINDLL test.cpp /MT /link /WHOLEARCHIVE:filamat.lib utils.lib shaders.lib smol-v.lib filabridge.lib /FORCE:MULTIPLE /DLL /OUT:filamat-shared.dll
By default _declspec(dllexport) is not defined for windows symbols, so when generating DLLs they don’t get exported. Defining UTILSPUBLIC in utils/compiler.h gets around this. Have to do this if we want the above cl.exe to actually output symbols in teh DLLs.
Add UTILSPUBLIC in many places.
INSTALL will put stuff in a directory nicely, but the exported headers can’t be used since it has dllexport. Use the original headers, but the updated lib
diff --git a/filament/include/filament/Box.h b/filament/include/filament/Box.h index 002a4ec0..9c3bdc5d 100644 --- a/filament/include/filament/Box.h +++ b/filament/include/filament/Box.h @@ -110,7 +110,7 @@ public: * @return the bounding box of the transformed box. * Result is undefined if \p m is not a rigid transform */ - friend Box UTILS_PUBLIC rigidTransform(Box const& box, const math::mat4f& m) noexcept; + friend Box rigidTransform(Box const& box, const math::mat4f& m) noexcept; /** * Computes the bounding box of a box transformed by a rigid transform @@ -119,7 +119,7 @@ public: * @return the bounding box of the transformed box. * Result is undefined if \p m is not a rigid transform */ - friend Box UTILS_PUBLIC rigidTransform(Box const& box, const math::mat3f& m) noexcept; + friend Box rigidTransform(Box const& box, const math::mat3f& m) noexcept; }; /** diff --git a/filament/include/filament/FilamentAPI.h b/filament/include/filament/FilamentAPI.h index 04885aa0..21ee83e5 100644 --- a/filament/include/filament/FilamentAPI.h +++ b/filament/include/filament/FilamentAPI.h @@ -60,7 +60,7 @@ public: * The actual implementation is in src/FilamentAPI-impl.h" */ template <typename T> -class UTILS_PUBLIC BuilderBase { +class BuilderBase { public: // none of these methods must be implemented inline because it's important that their // implementation be hidden from the public headers. diff --git a/filament/include/filament/IndexBuffer.h b/filament/include/filament/IndexBuffer.h index 25ea7739..c9830d11 100644 --- a/filament/include/filament/IndexBuffer.h +++ b/filament/include/filament/IndexBuffer.h @@ -60,7 +60,7 @@ public: UINT = uint8_t(backend::ElementType::UINT), //!< 32-bit indices }; - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: Builder() noexcept; diff --git a/filament/include/filament/IndirectLight.h b/filament/include/filament/IndirectLight.h index 105ee572..4a617f4e 100644 --- a/filament/include/filament/IndirectLight.h +++ b/filament/include/filament/IndirectLight.h @@ -97,7 +97,7 @@ class UTILS_PUBLIC IndirectLight : public FilamentAPI { public: //! Use Builder to construct an IndirectLight object instance - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: Builder() noexcept; diff --git a/filament/include/filament/LightManager.h b/filament/include/filament/LightManager.h index d3c6a5d5..aa1d7216 100644 --- a/filament/include/filament/LightManager.h +++ b/filament/include/filament/LightManager.h @@ -257,7 +257,7 @@ public: }; //! Use Builder to construct a Light object instance - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: /** diff --git a/filament/include/filament/Material.h b/filament/include/filament/Material.h index 607730d4..6be3a8df 100644 --- a/filament/include/filament/Material.h +++ b/filament/include/filament/Material.h @@ -83,7 +83,7 @@ public: Precision precision; }; - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: Builder() noexcept; diff --git a/filament/include/filament/RenderableManager.h b/filament/include/filament/RenderableManager.h index aab30455..ec6cc798 100644 --- a/filament/include/filament/RenderableManager.h +++ b/filament/include/filament/RenderableManager.h @@ -113,7 +113,7 @@ public: /** * Adds renderable components to entities using a builder pattern. */ - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: enum Result { Error = -1, Success = 0 }; diff --git a/filament/include/filament/Skybox.h b/filament/include/filament/Skybox.h index bea309e8..d453e677 100644 --- a/filament/include/filament/Skybox.h +++ b/filament/include/filament/Skybox.h @@ -66,7 +66,7 @@ class UTILS_PUBLIC Skybox : public FilamentAPI { public: //! Use Builder to construct an Skybox object instance - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: Builder() noexcept; diff --git a/filament/include/filament/Texture.h b/filament/include/filament/Texture.h index 97733865..babae594 100644 --- a/filament/include/filament/Texture.h +++ b/filament/include/filament/Texture.h @@ -100,7 +100,7 @@ public: //! Use Builder to construct a Texture object instance - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: Builder() noexcept; diff --git a/filament/include/filament/VertexBuffer.h b/filament/include/filament/VertexBuffer.h index a3f63ca0..cba561d8 100644 --- a/filament/include/filament/VertexBuffer.h +++ b/filament/include/filament/VertexBuffer.h @@ -59,7 +59,7 @@ public: using AttributeType = backend::ElementType; using BufferDescriptor = backend::BufferDescriptor; - class UTILS_PUBLIC Builder : public BuilderBase<BuilderDetails> { + class Builder : public BuilderBase<BuilderDetails> { friend struct BuilderDetails; public: Builder() noexcept; diff --git a/filament/src/Box.cpp b/filament/src/Box.cpp index 7acf7986..44235f0e 100644 --- a/filament/src/Box.cpp +++ b/filament/src/Box.cpp @@ -20,12 +20,12 @@ using namespace filament::math; namespace filament { -Box UTILS_PUBLIC rigidTransform(Box const& UTILS_RESTRICT box, const mat4f& UTILS_RESTRICT m) noexcept { +Box rigidTransform(Box const& UTILS_RESTRICT box, const mat4f& UTILS_RESTRICT m) noexcept { const mat3f u(m.upperLeft()); return { u * box.center + m[3].xyz, abs(u) * box.halfExtent }; } -Box UTILS_PUBLIC rigidTransform(Box const& UTILS_RESTRICT box, const mat3f& UTILS_RESTRICT u) noexcept { +Box rigidTransform(Box const& UTILS_RESTRICT box, const mat3f& UTILS_RESTRICT u) noexcept { return { u * box.center, abs(u) * box.halfExtent }; } diff --git a/filament/src/MaterialParser.h b/filament/src/MaterialParser.h index 1dfebc4e..d81c4741 100644 --- a/filament/src/MaterialParser.h +++ b/filament/src/MaterialParser.h @@ -41,7 +41,7 @@ namespace filament { class UniformInterfaceBlock; class SamplerInterfaceBlock; -class UTILS_PUBLIC MaterialParser { +class MaterialParser { public: MaterialParser(backend::Backend backend, const void* data, size_t size); diff --git a/filament/src/details/MaterialInstance.h b/filament/src/details/MaterialInstance.h index 2d307d06..0fb0d62c 100644 --- a/filament/src/details/MaterialInstance.h +++ b/filament/src/details/MaterialInstance.h @@ -36,7 +36,7 @@ namespace details { class FMaterial; -class UTILS_PUBLIC FMaterialInstance : public MaterialInstance { +class FMaterialInstance : public MaterialInstance { public: FMaterialInstance(FEngine& engine, FMaterial const* material); FMaterialInstance(const FMaterialInstance& rhs) = delete; @@ -62,7 +62,7 @@ public: } template <typename T, typename = is_supported_parameter_t<T>> - UTILS_PUBLIC void setParameter(const char* name, T value) noexcept; + void setParameter(const char* name, T value) noexcept; template <typename T, typename = is_supported_parameter_t<T>> void setParameter(const char* name, const T* value, size_t count) noexcept; diff --git a/libs/image/include/image/KtxBundle.h b/libs/image/include/image/KtxBundle.h index b3199e88..628fdc3e 100644 --- a/libs/image/include/image/KtxBundle.h +++ b/libs/image/include/image/KtxBundle.h @@ -18,14 +18,13 @@ #define IMAGE_KTXBUNDLE_H #include <math/vec3.h> -#include <utils/compiler.h> #include <cstdint> #include <memory> namespace image { -struct UTILS_PUBLIC KtxInfo { +struct KtxInfo { uint32_t endianness; uint32_t glType; uint32_t glTypeSize; @@ -37,7 +36,7 @@ struct UTILS_PUBLIC KtxInfo { uint32_t pixelDepth; }; -struct UTILS_PUBLIC KtxBlobIndex { +struct KtxBlobIndex { uint32_t mipLevel; uint32_t arrayIndex; uint32_t cubeFace; @@ -62,7 +61,7 @@ struct KtxMetadata; * * https://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/ */ -class UTILS_PUBLIC KtxBundle { +class KtxBundle { public: ~KtxBundle(); diff --git a/libs/image/include/image/KtxUtility.h b/libs/image/include/image/KtxUtility.h index 553280d6..8ec2859d 100644 --- a/libs/image/include/image/KtxUtility.h +++ b/libs/image/include/image/KtxUtility.h @@ -58,7 +58,7 @@ namespace ktx { * @param callback Gets called after all texture data has been uploaded to the GPU * @param userdata Passed into the callback */ - inline Texture* UTILS_PUBLIC createTexture(Engine* engine, const KtxBundle& ktx, bool srgb, + inline Texture* createTexture(Engine* engine, const KtxBundle& ktx, bool srgb, Callback callback, void* userdata) { using Sampler = Texture::Sampler; const auto& ktxinfo = ktx.getInfo(); @@ -85,7 +85,7 @@ namespace ktx { .format(texformat) .build(*engine); - struct UTILS_PUBLIC Userdata { + struct Userdata { uint32_t remainingBuffers; Callback callback; void* userdata; diff --git a/libs/image/src/KtxBundle.cpp b/libs/image/src/KtxBundle.cpp index 68ef89ac..300ab1cd 100644 --- a/libs/image/src/KtxBundle.cpp +++ b/libs/image/src/KtxBundle.cpp @@ -49,14 +49,14 @@ const uint8_t MAGIC[] = {0xab, 0x4b, 0x54, 0x58, 0x20, 0x31, 0x31, 0xbb, 0x0d, 0 namespace image { // This little wrapper lets us avoid having an STL container in the header file. -struct UTILS_PUBLIC KtxMetadata { +struct KtxMetadata { std::unordered_map<std::string, std::string> keyvals; }; // Extremely simple contiguous storage for an array of blobs. Assumes that the total number of blobs // is relatively small compared to the size of each blob, and that resizing individual blobs does // not occur frequently. -struct UTILS_PUBLIC KtxBlobList { +struct KtxBlobList { std::vector<uint8_t> blobs; std::vector<uint32_t> sizes; diff --git a/libs/utils/include/utils/compiler.h b/libs/utils/include/utils/compiler.h index 3013bd9d..87a8f030 100644 --- a/libs/utils/include/utils/compiler.h +++ b/libs/utils/include/utils/compiler.h @@ -33,7 +33,7 @@ #if __has_attribute(visibility) # define UTILS_PUBLIC __attribute__((visibility("default"))) #else -# define UTILS_PUBLIC __declspec(dllexport) +# define UTILS_PUBLIC #endif #if __has_attribute(packed)
This now links with VS 2017 applications.