| Naslov: | Gradient calibration in LSTM networks for enhanced learning efficiency : doctoral dissertation |
|---|
| Avtorji: | ID Tolić, Antonio (Avtor)
ID Skansi, Sandro (Mentor) Več o mentorju...  |
|---|
| Datoteke: |  DR_Tolic_Antonio_2026.pdf (2,62 MB)
MD5: BFB25993A6E7FD79ECC200BE9A511DE3
|
|---|
| Jezik: | Angleški jezik |
|---|
| Vrsta gradiva: | Doktorsko delo/naloga |
|---|
| Tipologija: | 2.08 - Doktorska disertacija |
|---|
| Organizacija: | FIŠ - Fakulteta za informacijske študije v Novem mestu
|
|---|
| Opis: | Recurrent Neural Networks (RNNs), most notably Long ShortTerm
Memory Networks (LSTMs), have established their efficacy across a wide range of sequential data tasks, especially in applications
demanding precise modeling of dependencies emerging from the inherent order of the data and complex behavior patterns. Despite substantial advances in the development of LSTM architectures, processing sequences with longterm dependencies remains nontrivial, as gradients may still vanish or grow to numerically unstable magnitudes when propagated across many time steps. In this context, a new approach to alleviating these difficulties is introduced, in which an LSTM architecture integrates Chrono Initialization (CI) with Layer Normalization (LN) to calibrate gradient propagation and more effectively support the learning of longrange dependencies. CI ensures that the gradients are neither too small nor too large, reducing the likelihood of both vanishing and exploding gradients and thereby enabling stable learning over long sequences. LN further contributes to robustness, leading to more consistent training dynamics and improved model performance across different sequence lengths and under varying input conditions, including shifts in data distribution and scale. The proposed approach was evaluated against LSTM baselines with and without CI applied to the forgetand inputgate biases. In
addition, several ablation variants were constructed to isolate the contribution of individual components of the proposed design. All model variants were evaluated on a diverse set of sequential
learning tasks, covering multiple task formulations and distinct hyperparameter settings. Throughout this evaluation, the proposed approach consistently demonstrated performance gains over all baselines, yielding greater predictive capability and lower validation loss. Additionally, the approach contributed to more efficient training, achieving faster convergence while preserving
strong generalization performance across different tasks and datasets. Its versatility was demonstrated on classification, regression, and sequence generation tasks. Overall, the proposed
enhancements improve longterm dependency modeling and yield more stable training dynamics in LSTMs, thereby addressing the aforementioned gradientrelated difficulties. Formal analysis
offers deeper insights into the underlying processes involved, thus establishing a robust basis for subsequent improvements in sequential data modeling. |
|---|
| Ključne besede: | chrono initialization, gradient propagation, layer normalization, long shortterm memory networks, recurrent neural networks |
|---|
| Status publikacije: | Objavljeno |
|---|
| Verzija publikacije: | Objavljena publikacija |
|---|
| Kraj izida: | Novo mesto |
|---|
| Kraj izvedbe: | Novo mesto |
|---|
| Založnik: | A. Tolić |
|---|
| Leto izida: | 2026 |
|---|
| Leto izvedbe: | 2026 |
|---|
| Št. strani: | XXI, 265 str. |
|---|
| PID: | 20.500.12556/ReVIS-14244 |
|---|
| UDK: | 004.85:004.032.26(043.2) |
|---|
| COBISS.SI-ID: | 284682499 |
|---|
| Opomba: | Na ov.: Doctoral dissertation;
|
|---|
| Datum objave v ReVIS: | 14.07.2026 |
|---|
| Število ogledov: | 70 |
|---|
| Število prenosov: | 5 |
|---|
| Metapodatki: |    |
|---|
|
:
|
Kopiraj citat |
|---|
| | | | Objavi na: |  |
|---|
Postavite miškin kazalec na naslov za izpis povzetka. Klik na naslov izpiše
podrobnosti ali sproži prenos. |
