Context. In recent years, conflicting results have provided an uncertain view of the dust-attenuated star-forming properties of z ≳ 4 galaxies. Aims. To solve this, we need to accurately measure the mean dust-attenuated properties of star-forming galaxies (SFGs) at 4 < z < 5 and therefore constrain the cosmic dust-attenuated star formation rate density (SFRD) of the Universe 1.3 Giga-years after the Big Bang. Methods. We used the deepest optical-to-near-infrared data publicly available in the Cosmic Evolution Survey (COSMOS) field to build a mass-complete (> 10 9.5 M ⊙ ) sample of SFGs at 4 < z < 5. Then, we measured their mean dust-attenuated properties (i.e., infrared luminosity, ⟨ L IR ⟩; dust-attenuated star formation rate, ⟨SFR IR ⟩) by dividing our sample in three stellar mass ( M * ) bins (i.e., 10 9.5 < M * / M ⊙ < 10 10 , 10 10 < M * / M ⊙ < 10 10.5 , and 10 10.5 < M * / M ⊙ < 10 11.5 ) and by stacking in the u v domain all archival Atacama Large Millimeter/submillimeter Array (ALMA) band 6 and 7 observations available for these galaxies. Then, we combined this information with their mean rest-frame ultraviolet (UV) emission measured from the COSMOS2020 catalog (i.e., UV luminosity, ⟨ L UV ⟩; UV spectral slope, ⟨ β UV ⟩; and unattenuated SFR, ⟨SFR UV ⟩), and constrained the IRX (≡ L IR / L UV )– β UV , IRX– M * , and SFR– M * relations at z ∼ 4.5. Finally, using these relations and the stellar mass function of SFGs at z ∼ 4.5, we inferred the unattenuated and dust-attenuated SFRD at this epoch. Results. SFGs follow an IRX– β UV relation that is consistent with that observed in local starbursts. Our measurements favors a steepening of the IRX– M * relation at z ∼ 4.5, compared to the redshift-independent IRX– M * relation observed at z ∼ 1 − 3. Our galaxies lie on a linear SFR– M * relation, whose normalization varies by 0.3 dex, when we exclude or include from our stacks the ALMA primary targets (i.e., sources within 3″ from the ALMA phase center). The cosmic SFRD( > M * ) converges at M * ≲ 10 9 M ⊙ , with SFGs at 10 8 < M * / M ⊙ < 10 9 contributing already less than 15% of the SFRD from all SFGs with M * > 10 8 M ⊙ . The cosmic SFRD at z ∼ 4.5 is dominated by SFGs with a stellar mass of 10 9.5 − 10.5 M ⊙ . Finally, the fraction of the cosmic SFRD that is attenuated by dust, SFRD IR (> M * )/SFRD(> M * ), is 90 ± 4% for M * = 10 10 M ⊙ , 68 ± 10% for M * = 10 8.9 M ⊙ (i.e., 0.03 × M ⋆ ; M ⋆ being the characteristic stellar mass of SFGs at this epoch) and this value converges to 60 ± 10% for M * = 10 8 M ⊙ . Conclusions. A non-evolving IRX– β UV relation suggests that the grain properties (e.g., size distribution, composition) of dust in SFGs at z ∼ 4.5 are similar to those in local starbursts. However, the mass and geometry of this dust result in lower attenuation in low-mass SFGs (≲10 10 M ⊙ ) at z ∼ 4.5 than at z ≲ 3. Nevertheless, the fraction of the cosmic SFRD that is attenuated by dust remains significant (∼68 ± 10%) even at such an early cosmic epoch.