| Title: | Gradient calibration in LSTM networks for enhanced learning efficiency : doctoral dissertation |
|---|
| Authors: | ID Tolić, Antonio (Author)
ID Skansi, Sandro (Mentor) More about this mentor...  |
|---|
| Files: |  DR_Tolic_Antonio_2026.pdf (2,62 MB)
MD5: BFB25993A6E7FD79ECC200BE9A511DE3
|
|---|
| Language: | English |
|---|
| Work type: | Doctoral dissertation |
|---|
| Typology: | 2.08 - Doctoral Dissertation |
|---|
| Organization: | FIŠ - Faculty of Information Studies in Novo mesto
|
|---|
| Abstract: | Recurrent Neural Networks (RNNs), most notably Long ShortTerm
Memory Networks (LSTMs), have established their efficacy across a wide range of sequential data tasks, especially in applications
demanding precise modeling of dependencies emerging from the inherent order of the data and complex behavior patterns. Despite substantial advances in the development of LSTM architectures, processing sequences with longterm dependencies remains nontrivial, as gradients may still vanish or grow to numerically unstable magnitudes when propagated across many time steps. In this context, a new approach to alleviating these difficulties is introduced, in which an LSTM architecture integrates Chrono Initialization (CI) with Layer Normalization (LN) to calibrate gradient propagation and more effectively support the learning of longrange dependencies. CI ensures that the gradients are neither too small nor too large, reducing the likelihood of both vanishing and exploding gradients and thereby enabling stable learning over long sequences. LN further contributes to robustness, leading to more consistent training dynamics and improved model performance across different sequence lengths and under varying input conditions, including shifts in data distribution and scale. The proposed approach was evaluated against LSTM baselines with and without CI applied to the forgetand inputgate biases. In
addition, several ablation variants were constructed to isolate the contribution of individual components of the proposed design. All model variants were evaluated on a diverse set of sequential
learning tasks, covering multiple task formulations and distinct hyperparameter settings. Throughout this evaluation, the proposed approach consistently demonstrated performance gains over all baselines, yielding greater predictive capability and lower validation loss. Additionally, the approach contributed to more efficient training, achieving faster convergence while preserving
strong generalization performance across different tasks and datasets. Its versatility was demonstrated on classification, regression, and sequence generation tasks. Overall, the proposed
enhancements improve longterm dependency modeling and yield more stable training dynamics in LSTMs, thereby addressing the aforementioned gradientrelated difficulties. Formal analysis
offers deeper insights into the underlying processes involved, thus establishing a robust basis for subsequent improvements in sequential data modeling. |
|---|
| Keywords: | chrono initialization, gradient propagation, layer normalization, long shortterm memory networks, recurrent neural networks |
|---|
| Publication status: | Published |
|---|
| Publication version: | Version of Record |
|---|
| Place of publishing: | Novo mesto |
|---|
| Place of performance: | Novo mesto |
|---|
| Publisher: | A. Tolić |
|---|
| Year of publishing: | 2026 |
|---|
| Year of performance: | 2026 |
|---|
| Number of pages: | XXI, 265 str. |
|---|
| PID: | 20.500.12556/ReVIS-14244 |
|---|
| COBISS.SI-ID: | 284682499 |
|---|
| UDC: | 004.85:004.032.26(043.2) |
|---|
| Note: | Na ov.: Doctoral dissertation;
|
|---|
| Publication date in ReVIS: | 14.07.2026 |
|---|
| Views: | 64 |
|---|
| Downloads: | 5 |
|---|
| Metadata: |    |
|---|
|
:
|
Copy citation |
|---|
| | | | Share: |  |
|---|
Hover the mouse pointer over a document title to show the abstract or click
on the title to get all document metadata. |
