Deleted jobs by TTL controller show application as "Out of Sync"

[v0l]

I have some jobs which use ttlSecondsAfterFinished to delete the job after completion, but now those jobs are showing as "Missing" because they are deleted.

Is this something that ArgoCD can detect and mark the application as in sync?

[chetan-rns]

Hi @v0l, how is the Job created? is it created by a Hook?

Hook deletion policies can be found here: at https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/#hook-deletion-policies

[v0l]

Hi @chetan-rns, its just a regular job, I tried using the hooks but found that those jobs run every sync which wasn't going to work for me.

I want the job to run only when it needs to, I.E when the spec changes it should sync and run the job again.

[todaywasawesome]

I don't think you answered how the job is created.

Also for the sync hooks, you can add boolean checks to your job to cancel it when not needed. Might be better?

[cmpilato]

I see the same thing. In my case the Job is defined as a resource:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: some-job
  namespace: some-namespace
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    argocd.argoproj.io/sync-options: Prune=false
spec:
  ttlSecondsAfterFinished: 30
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: some-service-account
      containers:
        ...

My job runs on each Sync, then 30 seconds after it finishes its pod is cleaned up and Argo starts to complain that about it being Missing and Out of Sync. How do I tell Argo that this is intentional?

[TalhaNaeem101]

I see the same thing. In my case the Job is defined as a resource:

@todaywasawesome @chetan-rns
It is correct that hooks are fine when we have to delete our jobs right after they are successful (in case of hook-succeed annotation). There are two deviations from this:
1- What if we need to keep our succeeded jobs for 30 minutes (as I need to see its logs). TTl option is suitable for this but argocd applications get outOfSync.
2- On every sync of the argocd application, the hooked job also runs, while there's no change in it's specs/config
I am looking for some solution for the first problem. If you wanna share any useful insights on this, please do share. would be highly grateful.

[djhoese]

Any other ideas for how to get around this issue? For any maintainers finding this, this is a standard way of having Job resources deployed and then cleaned up afterward:

https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/

I was hoping the IgnoreExtraneous would fix this for me, but it sounds like that didn't help you @cmpilato?

[cmpilato]

It did not.

[zopz]

I was able to get this working (for all intents and purposes) with the following:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded

When a sync is ran, it will spawn the job and remove it directly after. Unfortunately this won't respect the TTL defined in the job configuration, as Argo seems to only check the actual job status, but for me that was not a deal breaker.

@v0l

I want the job to run only when it needs to, I.E when the spec changes it should sync and run the job again.

I've unfortunately got nothing for you there, their hooks appear to be tied directly to the sync action itself, with no concept of resource state :-\

I agree that would be a great feature to have, though. Along with argocd.argoproj.io/hook-delete-policy respecting the ttlSecondsAfterFinished of the object it's looking to remove.

[djhoese]

Yes. Configure the Job in the helm chart to allow you to specify additional annotations via values.yaml. Then you can specify them in values.yaml and ArgoCD should see them and run the Job every time the Argo Application is Sync'd. You could read up on Argo hooks and have it run at different times, but that's up to you.

[devops-contribution]

Thanks @djhoese. In my case I have to add below annotations to make it work...
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded

One more question, will this work when we create resources using kustomize, please suggest. Thanks.

[djhoese]

No idea. I'm not a maintainer of ArgoCD and I don't use kustomize in my setup or know how it works.

[acuD1]

Hello everyones, i'm trying to do a Job that apply a migrations schema to the DB. I've tried everythings, and the Hooks seem's the way.
However the Hook is not triggered again with a rollback. Some of you have figured out how to resolve this ?

[djhoese]

This seems like an ArgoCD hook issue, not a TTL controller issue. I'd make a new issue or do another search on the documentation to see if running hooks on rollback is expected.

[TalhaNaeem101]

Hello community members,
This issue/bug needs to be resolved at high priority, also there are a lot of issues related to this, but not given some solution. Kindly can you please try fix this issue this is a blocker for us to use argocd for managing our beta workload.
⏫ ⚠️

[djhoese]

There is a workaround described above in a different thread by defining things as hooks (by adding annotations to your jobs).

Otherwise, why do community members need to fix this for you? Why does ArgoCD need to fix this for you? Are you paying for ArgoCD? Do you have developers of your own that can contribute a better fix for this?

[TalhaNaeem101]

Thank you for your comment @djhoese. No doubt it is a beautiful insight to get it fixed by our own developer; point taken.

But the problem, if I elaborate is that a solution is already mentioned on the argocd docs but it is not working as it is mentioned over there. I have added all the details here in a separate issue, please do get some of your time to see it. Thanks in advance.

[crenshaw-dev]

@TalhaNaeem101 I appreciate the detailed issue. Expecting responses within two hours is a pretty high bar, even for paid teams.

If the described behavior is in fact a bug, reproducing, diagnosing, and fixing it will take some developer's time. If they're not paid by your company, then those developers will work on their own timeline. Asking them to work faster, for free, is not appropriate.

If it's this critical/urgent, I would encourage you to ask your employer to dedicate some of its engineering resources to directly working on the Argo CD code base.

[TalhaNaeem101]

@crenshaw-dev Agreed, Awesome.
Actually, I am also trying to get it sorted out by our own devs, but definitely they will take more time to understand it and then fix it. Hence, I raised this ticker here as well, so that if the experts can do it, then it would be quite efficient. No intentions otherwise. Hope this clarifies it.
Rest assured, we together make the open-source contributions/projects grow by helping each other.
Thanks

[crenshaw-dev]

Understood, getting experts' opinions will definitely help your devs move faster in a new code base. Thanks for clarifying! 🙂

[seanorama]

This appears to still be an issue.

ArgoCD should be aware of how Kubernetes manages such resources.