InterDigital CE Patent Holdings Patente

The present principles generally relate to the storage and transmission of avatar texture animation data.

Specific syntax elements define a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message carrying information related to a tone mapping operation that can be applied as a post filter to the decoded image. This NNPFC SEI message may comprise information representative of the colour volume resulting from the tone mapping operation, information representative of characteristics of a target mastering display and information representative of adaptation parameters, related to the adaptation of the luminance level, quality level and the energy adaptation level that result from the tone mapping operation are provided. The information representative of the colour volume resulting from the tone mapping operation may also be carried using a second SEI message. An encoding method, a decoding method, an encoding apparatus and a decoding apparatus are described.

Some embodiments of a method may include initializing a three-dimensional (3D) Gaussian (3DG) generation process, wherein initializing the 3DG generation process is based on two or more temporally consecutive frames with views of a 3D environment; and performing the 3DG generation process to generate a representation of the 3D environment.

Some embodiments of a method may include: obtaining a scene description file corresponding to a shared scene environment; parsing the scene description file into one or more data structures; identifying elements of the shared scene environment that describe user interactions; determining which elements are related to only one user and which elements are related to multiple users; tracking, for each element related to multiple users, events referenced by the respective user interaction; detecting satisfaction of a multi-user condition corresponding to occurrence of the respective event by one or more users; and responsive to detecting the satisfaction of the multi-user condition, executing an action related to the identified element of the shared scene environment.

An encoding method is disclosed. An avatar animation unit (AAU) is obtained that comprises a header and a payload. From the header, it is determined whether the AAU is a keyframe AAU or a non-keyframe AAU. The AAU is finally encoded responsive to the determining.

Some embodiments of a method may include obtaining a MPEG Avatar Representation Format (ARF)-based file, wherein the MPEG ARF-based file comprises a list identifying one or more controller sets of an avatar, wherein each of the one or more controller sets comprise one or more controllers, and wherein each of the one or more controllers comprise a list of one or more channels; verifying the MPEG ARF-based file comprises information corresponding to the one or more channels; parsing the information corresponding to the one or more channels into one or more respective channel data structures; and controlling a component of an avatar using at least one of the one or more channel data structures.

A method comprising: obtaining (800) a current block of a picture; deriving (801) a first list of a first number of intra prediction modes; generating (802) a prediction of a template of samples in a neighborhood of the current block for each intra prediction mode of the first list; for each predictor, obtaining (803) a prediction error between the predictor and the template; obtaining (804) a second list of reference lines in a neighborhood of the current block and forming a TMRL list of a second number of template based multiple reference lines (TMRL) candidates by browsing the intra prediction modes of the first list in ascending order of the prediction error and browsing the reference lines of the second list from a nearest reference line to a furthest reference line, each TMRL candidate of the TMRL list comprising an intra prediction mode extracted from the first list during the browsing the first list and a reference lines extracted from the second list extracted during the browsing of the second list; and, determining (805) a TMRL predictor for the current block from the TMRL candidates of the TMRL list.

Specific syntax elements define a target colour volume resulting from some processing on pictures in a video coding system. An encoded video bitstream carries such syntax elements from an encoding device to a decoding device using a Target Colour Volume Information Supplemental Enhancement Information message or a Content Colour Volume Supplemental Enhancement Information message, thus allowing the encoding device to provide to the decoding device the target colour volume and thus adapt the processing accordingly. An encoding method, a decoding method, an encoding apparatus and a decoding apparatus are described.

A first indicator indicating that a prediction mode of a current block belongs to a list of most probable modes is decoded. A second indicator is then obtained that indicates that perspective prediction is to be applied to the current block. An index is also obtained that identifies a prediction mode in a list of most probable perspective modes derived from the list of most probable modes. The current block is reconstructed based on the prediction mode.

Systems, methods, and instrumentalities are configured for intra template matching prediction fusion. In examples, a video decoding device may be configured to determine that intra template matching prediction (ITMP) fusion is used for a video block. The device may obtain a selected block vector (BV) indication configured to indicate a selected BV candidate. The device may determine, based on the selected BV candidate, a set of BV candidates for fusion. The device may reconstruct the video block based on the determined set of BV candidates.

In various implementations, methods and devices are disclosed that indicate to the decoder a picture dimension to be used for the film grain synthesis, the picture dimension persisting for the coded video sequence (or for the coded layer video sequence) to which the FGC SEI message(s) is(are) associated. For instance, in one embodiment, a method of video decoding or rendering is disclosed that comprises obtaining a film grain picture dimension at which a film grain synthesis should be applied; obtaining a picture of a video sequence, the picture having a picture dimension; upon determining that the film grain picture dimension is different from the picture dimension, resampling the picture to the film grain picture dimension; and applying film grain synthesis to the resampled picture.

Methods and apparatuses for encoding/decoding a video are provided wherein at least two prediction blocks for a current block of an image of the video are obtained and wherein at least one of the at least two prediction blocks is obtained using a block vector-based prediction. The at least two prediction blocks are blended to provide a predicted block, wherein the blending uses blending weights determined from a template of the current block and the current block is encoded or decoded based on the predicted block.

Specific syntax elements define a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message carrying information related to a tone mapping operation that can be applied as a post filter to the decoded image. This NNPFC SEI message may comprise information representative of the colour volume resulting from the tone mapping operation and information representative of adaptation parameters, related to the adaptation of the luminance level, quality level and the energy adaptation level that result from the tone mapping operation are provided. The information representative of the colour volume resulting from the tone mapping operation may also be carried using a second SEI message. An encoding method, a decoding method, an encoding apparatus and a decoding apparatus are described.

Systems, methods, and instrumentalities are configured for intra template matching prediction fusion. In examples, a device (e.g., a video encoding and/or decoding device) may be configured to obtain, for a video block, a block vector (BV) candidate of a first fusion set. The device may compare a template matching (TM) cost of the BV candidate to a threshold associated with a first fusion set. The device may, based on the comparison, determine to bypass the BV candidate from the first fusion set. The device may determine a fused prediction block based on the first fusion set, a second fusion set, and the bypassed BV candidate. The device may decode the video block based on the fused prediction block.

Systems and method directed to point cloud deep learning inpainting of projections. One example is directed to a computer-implemented method for video encoding or decoding including deriving non-projected points from decoded geometry of a three-dimensional -3D-point cloud, predicting attributes for the non-projected points using a convolutional neural network, and encoding or decoding the 3D point cloud based on the attributes.

Systems, methods, and instrumentalities are disclosed for pruning latent variables for a hybrid implicit neural representation (INR). In examples, a decoding/encoding device may include a processor. The device may be configured to extract a set of features from an image. The processor may select, from a plurality of sets of predefined weights, a set of weights based on the extracted set of features. The processor may determine, based on the selected set of predefined weights, a probability distribution associated with a plurality of latent variables. The processor may entropy encode the plurality of latent variables based on the probability distribution.

A method, e.g., a decoding method, is disclosed. A set of latent variables representative of a current picture, parameters of a set of at least one first neural network, and at least one parameter of a downsampling chroma network are first obtained. The current picture is reconstructed based on the set of latent variables, the parameters of the set of at least one first neural network and the parameters of the downsampling chroma network.

A method (e.g., a decoding method) is disclosed. A set of latent variables representative of a current picture and neural network parameters are obtained. The current picture is reconstructed using the set of latent variables as input of a set of at least one first neural network parametrized by the neural network parameters. To this aim, an output of the first neural network is used as input of a forward pass of an invertible downsampling chroma neural network to obtain the reconstructed current picture.

Systems, methods, and instrumentalities are disclosed for entropy coding of on the border predicted probabilities. A device for video decoding or encoding may include obtaining an interval range represented by a number of bits that are greater than nine. The device may obtain a context adaptive binary arithmetic coding (CABAC) probability associated with a CABAC bin. The device may determine a range of a least probable symbol (LPS) based on the CABAC probability and the interval range. The device may entropy decode or encode the CABAC bin based on the range of the LPS. The device may determine the number of bits based on a range adjustment parameter. The range adjustment parameter may be any integer value of one to six.

Video coding (e.g., encoding and/or decoding) may be performed by optimizing a block vector search process. In examples, a device (e.g., a decoder for video decoding) may be configured to obtain a plurality of candidate block vectors (BVs) for a current video block. The device may determine a refinement area associated with a current BV from the plurality of candidate BVs. The refinement area may define BV refinement candidates for the current BV. The device may determine a refined BV, for example, based on a cost analysis. The cost analysis may be applied to (e.g., only) a subset of the BV refinement candidates defined by the refinement area. The device may add the refined BV to a candidate BV list for the current video block.

Systems, methods, and instrumentalities for signal-specific and universal dictionary based implicit neural representation. An example video decoding device may receive a coding block and video data associated with the coding block. The video data may indicate a signal-specific implicit neural representation (INR) dictionary, INR tail parameters, and a coefficient matrix. The device may determine a combined INR dictionary based on the signal-specific INR dictionary and a universal INR dictionary. The device may construct an INR based on the combined INR dictionary, the INR tail parameters, and the coefficient matrix. The device may reconstruct the coding block based on the INR.

A method and device use a processing algorithm to perform energy control of a content. The impact of the energy control may be determined, in relation with health, economics, pollution, raw material, social recognition or biodiversity. An indicator representing the impact of energy control may be displayed, allowing the user to evaluate values of these indicators in relation to the quality level obtained on the energy-controlled content.

In various implementations, methods and devices are disclosed that involves quantization aware training for dictionary-driven INR (Implicit Neural Representation). For instance, one or more quantization parameters for an INR network corresponding to an image region are obtained, wherein the INR network is reconstructed at both encoding and decoding by combining tail layers to head layers, the tail layers being defined by the one or more inverse quantized network parameters and the head layers being a weighted combination of one or more INR basis functions by the one or more inverse quantized coefficients, respectively. Then, the image region is reconstructed based on the reconstructed INR network. The one or more quantization parameters for the INR network, the one or more quantized coefficients of the head layers and the one or more quantized network parameters of the tail layers are encoded into or decoded from the bitstream.

In one implementation, a method of video decoding is presented, comprising: receiving at least a syntax element associated with energy adaptation; determining that one or more visual dimensions of a plurality of visual dimensions of a video are to be adjusted based on the at least a syntax element; and adapting a neural network based filter to adjust the one or more visual dimensions. At the encoder side, a method of video encoding is presented, comprising: determining that one or more visual dimensions of a plurality of visual dimensions of a video are to be adjusted for energy adaptation; and encoding at least a syntax element to indicate that the one or more visual dimensions are to be adjusted by a neural network based filter.

Systems, methods, and instrumentalities are disclosed for pruning latent variables for a hybrid implicit neural representation (INR). A device for video encoding may include a processor. The device may obtain a plurality of latent variables associated with an image. The device may identify a first portion of the plurality of latent variables and/or a second portion of the plurality of latent variables. The device may determine an INR network parameter based on the second portion of the plurality of latent variables. The device may encode the image based on the second portion of the plurality of latent variables and/or the INR network parameter. The device may generate a latent variable mask by analyzing the image. The device may apply the latent variable mask to the plurality of latent variables to identify the first portion of the plurality of latent variables and/or the second portion of the plurality of latent variables.

A device for video decoding may determine a range of values for a coordinate associated with a non-zero coefficient of a transform block (TB). The coordinate may indicate a sample location associated with the non-zero coefficient in the TB. The device may determine a context-adaptive binary arithmetic coding (CABAC) context for entropy decoding a coordinate indication indicating whether a coordinate suffix associated with the range of values may be received. The device may determine the coordinate based on the coordinate indication. The device may decode the TB based on the coordinate associated with the non-zero coefficient. The device may entropy decode the coordinate indication associated with the coordinate based on the CABAC context. The device may determine that the coordinate suffix is not to be received and/or that the coordinate is to be determined based on a subset of the range values and/or the size of the TB.

Systems, methods, and instrumentalities are disclosed for transform set selection based on template cost analysis. For example, a device (e.g., a video encoding device and/or a video decoding device) may obtain at least one prediction sample. The device may obtain a histogram of gradient (HoG). The device may determine whether to perform a template cost analysis. Based on a determination to perform the template cost analysis, the device may obtain at least one intra mode candidate based on the HoG. The device may determine a respective template cost associated with at least one intra mode candidate. The device may reorder at least one intra mode candidate. The device may obtain at least one virtual intra prediction mode (VIPM). The device may obtain a transform set and obtain a transform kernel. The video decoding device may decode the current block, and the video encoding device may encode the current block.

Systems, methods, and instrumentalities are disclosed for a video decoding and encoding device. A device may include a processor configured to determine that intra template matching prediction (intraTMP) is enabled for a video block. The processor may be configured to determine that intraTMP is used for decoding a first adjacent block of the video block. The processor may be configured to determine, based on a local illumination compensation (LIC) indication of the first adjacent block, a context-adaptive binary arithmetic coding (CABAC) context associated with entropy decoding a LIC indication associated with the video block. The processor may be configured to entropy decode the LIC indication associated with the video block based on the CABAC context. The processor may be configured to decode the video block based on the LIC indication.

Some embodiments of a method may include: obtaining mesh data and images of an avatar head captured by input cameras; initializing parameters of a plurality of three-dimensional (3D) Gaussians from the mesh data; modifying a local density of the 3D Gaussians adaptively to values of their respective parameters; processing the plurality of 3D Gaussians using one or more Gaussian update processes; obtaining camera viewpoints for one or more of the input cameras; generating a plurality of 2D Gaussians by performing a two-dimensional (2D) projection of the plurality of processed 3D Gaussians to an image plane for each of the one or more input cameras using the respective camera viewpoints; generating an output image by performing a differentiable rasterization of the plurality of 2D Gaussians; and updating the parameters of the 3D Gaussians and parameters of the one or more Gaussian update processes.

Apparatuses and methods are described including techniques for processing color data of a video. Techniques include obtaining an input color segment containing colors of collocated video pixels. In a first color space, a set of color segments is searched to find segments that consume less power than that consumed by the input color segment. If such a set of color segments is found, to preserve energy, the input color segment is replaced with the closest color segment from that set. If such a set of color segments is not found, in a second color space, a color segment is determined that minimizes power consumption and the distance to the input color segment. Then, to preserve energy, the input color segment is replaced with the determined color segment providing that it consumes less power than that consumed by the input color segment.

Some embodiments of a method may include: obtaining avatar mesh data and avatar images captured from different viewpoints associated with an avatar; obtaining a plurality of processed 3D Gaussians fitted to the avatar mesh data and avatar images; determining geometric parameters associated with the plurality of processed 3D Gaussians; determining appearance parameters associated with the plurality of processed 3D Gaussians; populating an avatar data structure associated with the geometric and appearance parameters; encoding the avatar data structure to generate Avatar JSON Interchange File (AJIF) data; and transmitting the encoded AJIF data to a device.

The systems, methods and devices are configured to use the notion of haptic illusions and convert haptics to at least one of audio, image and video. Generally, the conversions occur when an application does not have the appropriate haptic device, or even does not have any haptic device to play haptics. Instead of not playing the haptic tracks, a conversion is performed to at least one of audio and video to enable the system to provide the illusion of haptics feedback. These systems, methods and devices may be applicable in media applications, such as gaming, movies, music, XR, communication, for example.

Methods, apparatus and data stream are provided for encoding and decoding a three-dimensional (3D) point cloud. The 3D point cloud is processed to obtain a coded geometry, a reconstructed geometry obtained by decoding the coded geometry and adapted attributes. The adapted attributes are obtained by tracking indices of points of the point cloud or by transferring attributes to the reconstructed geometry. The 3D point cloud is divided in a regular grid of blocks, projecting adapted attributes of points of each non-empty block on inner faces of the blocks and selecting one of the inner faces to obtain patches. The patches are packed in an atlas image according to a scanning process and the atlas image is encoded in a data stream with metadata indicating selected inner faces.

A method comprising obtaining (700) two consecutive pictures of a video sequence; computing (701) a first histogram of a first of the two consecutive pictures and a second histogram of a second of the two consecutive pictures; obtaining (702) a plurality of weighting curves, each weighting curve weighting a value of each bin of an histogram in function of a bin number; for each weighting curve of the plurality: applying (704) the weighting curve to the first and the second histograms to obtain weighted first and second histograms; summing (705) values of bins of the weighted first histogram to obtain a first sum and summing values of bins of the weighted second histogram to obtain a second sum; obtaining (706) a ratio of sums from the first sum and the second sum; and, calculating (707) an evidence value from the first sum, the second sum and the ratio; and, determining (708) that a scene cut occurs between the first and the second pictures based on at least one of the evidence values.

Systems and method directed to spanning intra prediction mode information using reconstructed coding unis. One example is directed to a computer-implemented method for video encoding or decoding including deriving intra prediction mode -IPM- information from one or more reconstructed samples of a coding unit, storing the IPM information in an IPM buffer corresponding to the coding unit, and encoding or decoding a picture including the coding unit based on the IPM buffer.

Systems, methods, and instrumentalities are disclosed for using a neural network (NN)-based film grain (FG) analysis. For example, a device may include a processor that is configured to perform one or more of the following. The device (e.g., a video processing device) may obtain a grainy video. The grainy video may be a video source with film grain. The device may feed the grainy video to a film grain analysis neural network (FGA-NN) module. The FGA-NN module may include a feature extraction network (e.g., a backbone) and at least one task-specific head. At least one task-specific head may be configured for a prediction task. For example, the prediction task may include a classification task and/or a regression task. The device may obtain at least one film grain characteristic (FGC) parameter based on an output of the FGA-NN module.

Methods and apparatus are provided for performing the quantization constrained (QC) correction of a given transform coefficient cf associated to a quantization level cq when the minimum and maximum bound offsets are individually enabled or disabled. In one embodiment, signaling methods are provided to save signaling cost of bound offset parameters if the corresponding bound offset is disabled. Another embodiment provides performing QC correction of a transform coefficient using individually enabled or disabled bound offsets. In another embodiment, computing both the minimum and the maximum bound offsets is performed using a single bound offset signaled in the bitstream.

Systems, methods, and instrumentalities are disclosed for film grain synthesis frequency shaping. As described herein, a device may include a processor that is configured to perform one or more of the following. The device may obtain a model parameter set. The model parameter set may be, or may include, a cutoff frequency. Based on the model parameter set (e.g., the cutoff frequency), the device may generate a soft frequency window. The soft frequency window may be at least one of a 2D low-pass window, a separable 2D low-pass window, or a low-pass 20 frequency window. The device may generate a synthetic grain based on the soft frequency window. For example, the device may generate the synthetic grain based on (e.g., applying) an inverse transform of the soft frequency window. The inverse transform may be associated with a discrete cosine transform (DCT).

Methods and apparatus are provided for signaling of a bound offset used in the quantization correction and defined from an integer bound offset parameter O_i and an integer shift parameter S_i, as the ratio oi2Si. The proposed methods apply the same way for the signaling of either a minimum bound offset o_min (i.e., i=0) or a maximum bound offset o_max (i.e., i=1). In one embodiment the integer bound offset parameter is signaled using its full bitdepth N_i. In a second embodiment, the integer bound offset parameter is signaled using a reduced bitdepth M_i and associated mapping parameters. In a second embodiment, signaling or inferring a flag indicates that a default value of the integer bound offset parameter is used.

Some embodiments of a method may include obtaining a bitstream of immersive video content captured by a combination of at least a first camera and a second camera; obtaining a first set of camera parameters corresponding to the first camera; obtaining a second set of camera parameters corresponding to the second camera; adapting one or more displacement sensitivity parameters associated with the immersive video content based on the first and second set of camera parameters; and rendering the immersive video content using the adapted one or more displacement sensitivity parameters.