Learning Continuous Neural Representations for Scalable and High-fidelity Electron Microscopy (ENCODE)

ENCODE is a general framework that redefines electron microscopy (EM) data storage by moving beyond discrete, frame-based pixel arrays. By learning the underlying continuous signal from raw data, ENCODE replaces bulky image stacks with a compact neural network representation. This approach provides a scalable pathway for high-fidelity storage and robust inference of physical dynamics from undersampled measurements.

🚀 Getting Started

Installation

1. Clone the repository:

   git clone https://github.com/TISGroup/ENCODE.git

2. Prepare the Conda environments:

   cd ENCODE
   conda create -n encode python=3.10
   conda activate encode
   pip install -r requirements.txt

💡 Usage

1. Download Dataset

To get started, first download the dataset and place it in the data directory.

Example datasets are available on Zenodo.

Note: If you want to use your own data, ensure it is in .tif or .npy (for 4D-STEM) format and update the data name in the relevant run_*.sh script.

2. Description of hyperparameters

General

• num_frames: The number of video frames to process. A value of -1 indicates that all available frames should be used.
• frame_gap: The temporal distance between frames selected from the video. A value of 1 means consecutive frames are selected.
• is_4dstem: Whether the input image type is 4D-STEM.
• input_bit: The bit depth of the input data.

Model

• embed_dim: The dimension for the embedding layers in the ConvNeXt encoder.
• fc_hw_dim: The height, width, and dimension of the latent feature.
• act: The activation function used in the model.
• stride_list: A list of stride values for each decoding block.
• t_dim: A list of temporal dimensions for the multi-resolution grids.
• expansion: A factor used to expand the number of channels in the first stage of the decoder.
• reduction: A factor used to reduce the number of channels in subsequent stages of the decoder.
• lower_width: The minimum number of channels allowed in the decoder layers. The channel width will not drop below this value.
• conv_type: The type of convolutional operation used for upsampling in the decoding blocks.
• out_channel: The number of output channels. A value of 1 for grayscale images.
• wavelet_levels: The number of wavelet decomposition levels used by the DWT layer to process the input images.

3. Compress

• Compress simulated STEM data

  bash scripts/train_stem.sh

• Compress experimental TEM data

  bash scripts/train_tem.sh

• Compress experimental 4D-STEM data

  bash scripts/train_4dstem.sh

🙏 Acknowledgments

This project builds upon the foundational work of HNeRV, E-NeRV, FFNeRV and HiNeRV. We appreciate their efforts in advancing this field and for making their work publicly available.